R: [Developers] Parallelization of computations

[Kristofer Tingdahl]

Dear Corrao,
> regarding parallelism, I guess most of my plugins can not exploit 'independent samples' multithreading the way the attached API would allow; nonetheless, any two distinct <il,xl> locations are independent, so I could imagine computeData being called for distinct traces in separate threads.
>   
The reason we split the traces in two was simply that it made buffering 
easier - both threads would access the same input data, and it was easy 
to know what should buffered in memory. Another reason was that the 
fileformat required traces to be written in the correct order. We are 
aware that this is a weakness in the attribute engine, and we have 
considered changing it so that there is one active calculation engine 
for each thread, with separate buffers. We have also considered having N 
threads compute every Nth thread with the same buffer.

We think the latter is the optimum approach, but that requires 
significant rewriting of the engine including all dGB and third party 
attributes. We have therefor put this on the shelf for now, but it will 
be one of the specs whenever we make a major revision of the engine.

The approach to split the the area in blocks is utilized in the 
multi-machine processing. Current version of OD does however only send 
one block to each host, and the host does not divide it internally.

I believe this is something we could change in the process_attrib 
executable.
> Said that, I noticed that process_attrib does not spawn multiple threads (they should show up as distinct processes in ps -efwww) and top reports that the two CPUs are not contributing equally: one is idle and one 99% busy.
>   
The process_attrib should spawn into the number of processors that you 
have on the calculation machine. Having said that, it still divides the 
traces, so if the attribute won't do that, the calculation won't be 
parallelized.

> Now, is it a platform issue (OS, kernel version, etc..) or just that multithreading is not implemented (yet) the way I would suggest? 
>   
process_attrib will use the number of processors that are set in 
$HOME/.od/settings. If no setting is found there, it will use the number 
of processors the the OS reports. An attribute like the steering cube 
does for example scale perfectly on both 2 and 4 processors (I don't 
have any 8 processor machine around here to test more).

We are FYI developing a simplified attribute engine, called the 
VolumeProcessing engine for attributes that have dependencies in all 
directions. In the VolumeProcessing engine, we form a chain of 
processes. Each process is has one input volume and one output volume, 
both in memory, and the process has one 'process' function that may go 
through the data any way it wants in one or may steps, with any number 
of threads. The VP is currently used in the Velocity Model Building 
project, and we have not decided how to integrate it in OD Base yet.

Regards,


Kristofer

-- 
Dr. Kristofer Tingdahl
dGB Earth Sciences
1 Sugar Creek Center BLVD #935; Sugar Land, TX 77478; USA
+1 281 240 3939