openmolecules.org Forum: Functionality » Can DW handle big data?

Home » DataWarrior » Functionality » Can DW handle big data? (multithreading, low cost of memory)

Show: Today's Messages :: Polls :: Message Navigator

Re: Can DW handle big data? [message #688 is a reply to message #682]

Thu, 24 October 2019 20:59

thomas
Messages: 729
Registered: June 2014

Senior Member

Dear DaRong,

3 million rows is already a lot. I recommend for very large files to use DataWarrior on Linux, because you can easily increase the memory maximum that there is at least no memory problem. DataWarrior uses multithreading for most functions, which benefit from it. However, reading a file is a serial process and cannot easily be parallelized. Possibly I could gain some performance, when distributing the data analysis after file loading on multiple cores. I will put it on the agenda, but not before the next release, which I anticipate before the end of the year.

Thanks and best wishes,

Thomas

Report message to a moderator

[Message index]

		Can DW handle big data? By: greatzdl on Tue, 22 October 2019 08:00
		Re: Can DW handle big data? By: thomas on Thu, 24 October 2019 20:59
		Re: Can DW handle big data? By: nbehrnd on Sun, 27 October 2019 17:01

Previous Topic:	Synchronise Colours
Next Topic:	Exclude function in the structure filter does not work with 2 exclude groups

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Apr 20 02:55:41 CEST 2025

Total time taken to generate the page: 0.00672 seconds