R: R: [Developers] Parallelization of computations

[Corrao Antonio]

Hi,
I did reconsider some of my plugins in the light of your answer: actually some of them could conform to opendtect current task model, except for the following aspect: a trace is split in N blocks which can be elaborated independently, but each block computes a slightly overlapping segment of the output, to avoid border effects. Therefore, after each thread has produced its result, the output should be computed summing each of them with proper tapering and cross-fading.

Graphically:
output task 1: _******__________ +
output task 2: _____******______ +
output task 3: _________******__ +

output attrib: __*************__ =

If I do allow each thread to directly update the output vector, I suspect thread safety is compromised when two threads are changing the same output[k] value. 
I guess I have to add a method to the AttribProvider class that is called after all threads have completed their work, to finalize the attrib output.

Hope I manage to do it just deriving a class from AttributeProvider and adding some kind of afterProcessingDone() method, but I haven't examined the code in detail.

Do you have any suggestions?

Best Regards,
AC



Antonio Corrao
Senior Geophysicist
GEOS dept. - Eni E&P Division
V. Emilia 1 - 20097 San Donato Milanese - Italy
Tel:+39.02.520.61150 - Fax:+39.02.520.63897 - mailto:antonio.corrao at eni.it
_____________________________________________________________________________
Eni S.p.A. - Sede legale in Roma, Piazzale Enrico Mattei 1, 00144 Roma - Cap.sociale 4.005.358.876,00 i.v.
Registro Imprese di Roma - Codice Fisc. 00484960588 - P.IVA 00905811006 - R.E.A. Roma n.756453


> -----Messaggio originale-----
> Da: developers-bounces at opendtect.org [mailto:developers-
> bounces at opendtect.org] Per conto di Kristofer Tingdahl
> Inviato: lunedì 3 marzo 2008 23.43
> A: developers at opendtect.org
> Oggetto: Re: R: [Developers] Parallelization of computations
>
> Dear Corrao,
> > regarding parallelism, I guess most of my plugins can not exploit
> 'independent samples' multithreading the way the attached API would allow;
> nonetheless, any two distinct <il,xl> locations are independent, so I
> could imagine computeData being called for distinct traces in separate
> threads.
> >
> The reason we split the traces in two was simply that it made buffering
> easier - both threads would access the same input data, and it was easy
> to know what should buffered in memory. Another reason was that the
> fileformat required traces to be written in the correct order. We are
> aware that this is a weakness in the attribute engine, and we have
> considered changing it so that there is one active calculation engine
> for each thread, with separate buffers. We have also considered having N
> threads compute every Nth thread with the same buffer.
>
> We think the latter is the optimum approach, but that requires
> significant rewriting of the engine including all dGB and third party
> attributes. We have therefor put this on the shelf for now, but it will
> be one of the specs whenever we make a major revision of the engine.
>
> The approach to split the the area in blocks is utilized in the
> multi-machine processing. Current version of OD does however only send
> one block to each host, and the host does not divide it internally.
>
> I believe this is something we could change in the process_attrib
> executable.
> > Said that, I noticed that process_attrib does not spawn multiple threads
> (they should show up as distinct processes in ps -efwww) and top reports
> that the two CPUs are not contributing equally: one is idle and one 99%
> busy.
> >
> The process_attrib should spawn into the number of processors that you
> have on the calculation machine. Having said that, it still divides the
> traces, so if the attribute won't do that, the calculation won't be
> parallelized.
>
> > Now, is it a platform issue (OS, kernel version, etc..) or just that
> multithreading is not implemented (yet) the way I would suggest?
> >
> process_attrib will use the number of processors that are set in
> $HOME/.od/settings. If no setting is found there, it will use the number
> of processors the the OS reports. An attribute like the steering cube
> does for example scale perfectly on both 2 and 4 processors (I don't
> have any 8 processor machine around here to test more).
>
> We are FYI developing a simplified attribute engine, called the
> VolumeProcessing engine for attributes that have dependencies in all
> directions. In the VolumeProcessing engine, we form a chain of
> processes. Each process is has one input volume and one output volume,
> both in memory, and the process has one 'process' function that may go
> through the data any way it wants in one or may steps, with any number
> of threads. The VP is currently used in the Velocity Model Building
> project, and we have not decided how to integrate it in OD Base yet.
>
> Regards,
>
>
> Kristofer
>
> --
> Dr. Kristofer Tingdahl
> dGB Earth Sciences
> 1 Sugar Creek Center BLVD #935; Sugar Land, TX 77478; USA
> +1 281 240 3939
>
>
> _______________________________________________
> Developers mailing list
> Developers at opendtect.org
> http://lists.opendtect.org/mailman/listinfo/developers

------------------------------------------------------------------------------------------------------------------------------------------
Message for the recipient only, if received in error, please notify the sender and read http://www.eni.it/disclaimer/