openmolecules.org Forum: Functionality » Why is clustering limited up to 10000 molecules?

Home » DataWarrior » Functionality » Why is clustering limited up to 10000 molecules?

Show: Today's Messages :: Polls :: Message Navigator

Why is clustering limited up to 10000 molecules? [message #75]

Wed, 22 July 2015 16:37

Nastasia
Messages: 3
Registered: July 2015

Junior Member

I am trying to cluster my molecules, but the number of my molecules is far beyond 10000, actually it's about 50000. However, there are some duplicate molecules, which may be joined, but when I do that I loose my different descriptors important for each case, as they are just merged into some multiple group. Is it possible to increase the size for clusters?

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #78 is a reply to message #75]

Thu, 23 July 2015 21:40

thomas
Messages: 731
Registered: June 2014

Senior Member

Clustering is a very old functionality and not used very often. The used algorithm is reproducible and is not based on random starting point as other faster algorithms. The flipside, however, is that is not a particularly fast one and its memory consumption and processor usage grow quadratic with the size of the data file. Without a limit people would experience error messages or extreme processing times causing lots of frustration. I am aware that DataWarrior needs an alternative clustering algorithm and it is on the feature list for the future, but till now I considered clustering low priority, because often there are better methods to reach a certain goal.

Kind regards, Thomas

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #154 is a reply to message #78]

Sun, 28 February 2016 19:03

avkitex
Messages: 2
Registered: February 2016

Junior Member

Dear autors. What if I would like to cluster ~100000 compounds?
I understand that it could consume huge amount of time and ram.

Best,
Nikita

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #157 is a reply to message #154]

Thu, 03 March 2016 21:49

thomas
Messages: 731
Registered: June 2014

Senior Member

Dear Nikita, the current DataWarrior version is limited to 10.000 compounds. For 100.000 compounds the current algorithm would need 20GB of memory and many hours to finish. The only way to do it would be to patch the datawarrior.jar file (OSX or Linux only) or to change the source code and to recompile.

Regards, Thomas

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #158 is a reply to message #157]

Thu, 03 March 2016 21:54

avkitex
Messages: 2
Registered: February 2016

Junior Member

Dear, Thomas

I have servers with about 30Gb ram and a lot of time. And I woult like to try doing it.
Can you tell me how to patch and recompile jar file? And what is the best way to run it in console mode? (I have molecules in sdf or mol2 format. I would like to have a clusterisation tree in newick or compatable format.)

Looking forward to hear from you.

Best,
Nikita

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #159 is a reply to message #158]

Fri, 04 March 2016 22:24

thomas
Messages: 731
Registered: June 2014

Senior Member

Dear Nikita,

if you use Linux or OSX, I will send you a customized datawarrior.jar file without limitation. But don't complain, if the clustering process seems to take forever...

If your server runs Linux, you can install the datawarrior directory on the server and login from remote via 'ssh -X' and then launch datawarrior the usual way. Then you all DataWarrior windows open on the client machine while it executes on the server and uses server RAM and server cores.

Best wishes, Thomas

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #252 is a reply to message #159]

Wed, 22 March 2017 11:34

dataviz
Messages: 5
Registered: November 2016

Junior Member

Hello
it is indeed true that to be able to run the cluster function on a dataset larger than 10,000 would be really great considering all the new data available today... This can be done externally but Warrior is so convenient that in a future release, it really would be nice...
Sincerely

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #255 is a reply to message #252]

Thu, 30 March 2017 20:35

thomas
Messages: 731
Registered: June 2014

Senior Member

Hi Dataviz,

I have lifted the limit from 10.000 to 100.000 structures, but display a warning above 20.000 compounds. Nevertheless, clustering of significantly more structures than 20.000 will require a Linux or Macintosh computer, where it is easy to increase the memory maximum that DataWarrior is allowed to use. The change will be available with the next update.

Regards, Thomas

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #437 is a reply to message #255]

Thu, 10 January 2019 21:46

E3ubiquitinligase
Messages: 2
Registered: January 2019

Junior Member

How do we manually increase the memory and compound limit for Datawarrior on a Mac? The new tSNE feature is working unbelievably well and I'd like to be able to apply it to >100k molecules. It hovers around 2.46 GB memory while crunching the tSNE so I'd love to be able to reove the existing limits.

Many thanks!
Mike

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #438 is a reply to message #437]

Fri, 11 January 2019 00:00

thomas
Messages: 731
Registered: June 2014

Senior Member

on the Mac this is easy. You need to increase the -Xmx setting in the Info.plist file. The details are explained in the manual: on the bottom of this page
http://www.openmolecules.org/help/installation.html#Installa tion

Thomas

[Updated on: Fri, 11 January 2019 00:02]

Report message to a moderator

Re: Why is clustering limited up to 10000 molecules? [message #439 is a reply to message #438]

Fri, 11 January 2019 07:21

E3ubiquitinligase
Messages: 2
Registered: January 2019

Junior Member

Works wonderfully - thank you!!

Mike

Report message to a moderator

Previous Topic:	X = Y line in plot
Next Topic:	Clear category filters

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Jun 15 18:07:25 CEST 2025

Total time taken to generate the page: 0.07003 seconds