Parallella Community • View topic - Inter-process communication on the Parallella.
Inter-process communication on the Parallella.
OK, given a set of processor cores each running APL, how do we communicate between them ?
A conventional APL system will offer a range of communication processes, now usually including object-oriented methods of accessing .net, Java, R, etc. For AplX see http://www.microapl.co.uk/apl/aplxv5.html We need to find a set of communication methods that compactly gives us rapid communication between processes running on the cores of a Parallellla.
b) piping data out of APL through a pipeline of shell commands and back into APL. A new mechanism Pipe.
c) spawning a new long-lived process and connecting as many pipes to it as needed. A new mechanism Spawn.
2) Another possible mechanism is a global APL variable handling program, that local APLs can make read and write request to. We wish to preserve the rigorous APL way of knowing the full specification of the data : its size, shape and type, and this information is kept by the global program.
3) Alternatively it is possible to just share a variable such that a number of processes share the data but the data is kept by the process with the most recent data used to write the variable. Any particular read then accesses the data (and its type and structure information) from this, so a data read is then a local to local process transfer over the global interconnect. This still needs to be 'policed' by a process with a global overview.
Techniques 2 and 3 are similar to the SVC Shared Variable Control in AplX.
Your thoughts on these and other possible mechanisms are most welcome.
If you have the knowledge / ability / time to implement any of these using the local/shared memory system on the Parallella, that would be most welcome.
Re: Inter-process communication on the Parallella.
I expect you have been reading the Architecture manual already but here are some thoughts.
Reads are much slower than writes. A read goes on a slow link to the destination core then the data are returned by a write back to the originator. If you want to transfer data quickly have one core write it to another's memory, not by a core reading another's memory.
The DMA engines are good at transferring data around memory while minimising the impact of the CPU to do other things. (Best if DMA access is to 4K block of RAM not currently being used by CPU if possible.) There is a small overhead in setting up the DMA so it is not suited to tiny amounts of data.
Pipes naturally lend themselves to writes by the originator which is convenient. I'd outline a pipe mechanism as follows.
Each core that can be a destination for a pipe should maintain a list of available RAM areas in its local memory. It should use a mutex variable present in its local memory while maintaining the data structures describing the list. The mutex will be quick to access as it's local.
When a core wants to pipe to another core it acquires a mutex on the destination core's free area list and acquires a suitable region of RAM updates list, releases mutex.
Sending core sets up DMA to read data from its local memory and write to destination core's memory. DMA is started.
Sending core does other things. Destination core carries on obliviously.
Destination core looks for more work and notices sending core's notification in its work list.
Yes I had noticed that the DMA channels looked like important channels, but had got no further.
I will be brutally honest and say that I have hopes that someone or someones (as knowledgeable as yourself), will take it upon themselves to build pipe mechanisms. Whether tried and tested means, or more novel means particularly appropriate to these processors, will be the way forward, I do not know.
I will have quite a bit of my time taken in just getting the Aplc basically up and running to the best in such a shared memory context, where the size, shape and type of the data is preserved in the transfer. But pipe data can (initially anyway) be plain text streams of arbitrary length.
But certainly the first things that are needed are simple standard in and standard out pipe communication to the host processor.
I am trying to hack a simple fixed buffer substitute, but I am sure someone who reads this has better tools in their kit-box.
I haven't had a chance to look real in depth yet on the Parallella, but are hardware mutexs available? A mutex per core would make any sort of inter-core communication much easier, as any internal or external resource could "lock" a core for use. I could imagine this leading to issues if a core got frozen, but have a watchdog per core that would auto reset could probably get around this problem.
It is also a question of how DMA or a process end on another core would grab the mutex if a different core already has the lock. If multiple cores are all vying for access to the same process, this could likely get very interesting, which shows the need for the high-speed FPGA to manage these things. Maybe support a table of mutexes in the FPGA?
I feel like I'm missing something that is probably in the hardware design already though.
Mutexes can be done by the TESTSET instruction on the Epiphany chip. The memory location must be on same chip as the core executing it. Can't be on external DRAM. Haven't seen anything to suggest the FPGA can do this to Epiphany internal RAM so would assume for time being not possible..
timpart wrote: Mutexes can be done by the TESTSET instruction on the Epiphany chip. The memory location must be on same chip as the core executing it. Can't be on external DRAM. Haven't seen anything to suggest the FPGA can do this to Epiphany internal RAM so would assume for time being not possible..
Note that the mutex operations are encapsulated in the mutex e-lib family of functions.
You are right in that the mutex location has to be on-chip, and is accessible (for now) only from an eCore.
Would be good if one day we could get a mutex across both chips somehow. At the moment needs careful keeping out of each others way in the shared external DRAM unless I've missed something.
(I have tried it done various ways, sorry about the messy code, it is just for testing).
0: Message from eCore 0x8ca ( 3, 2): "tEstAplctext"
1: Message from eCore 0x84b ( 1, 3): "tEstAplctext"
2: Message from eCore 0x84b ( 1, 3): "tEstAplctext"
3: Message from eCore 0x888 ( 2, 0): "tEstAplctext"
4: Message from eCore 0x849 ( 1, 1): "tEstAplctext"
0: Message from eCore 0x8ca ( 3, 2): "tEst"
1: Message from eCore 0x84b ( 1, 3): "tEst"
2: Message from eCore 0x84b ( 1, 3): "tEst"
so I can call sprintf repeatedly, and build up a page or so of text.
Can anyone tell me how to separate these ?
Currently I am calling the legacy.ldf and this also needs to be changed so the aplc program is in internal memory.
I can do no more until these are sorted.
Last edited by Dr.BeauWebber on Mon Jun 03, 2013 10:03 pm, edited 1 time in total.
The problems I was having, getting simple output this way, have, as you can see above, now been sorted.
What we now need to consider are more advanced methods (with proper handshake/synchronisation) of communicating between separate aplc processes on different cores, and between these and the host. processor.
Aplc has, if one wishes, the ability to declare the variable type.
that the storage for this variable is given in some other place.
This offers a good way of having an overall process that specifies the correct sharing of global memory between the cores, and between cores and external memory, and the more localised memory for one particular core.