   [java] Speeding up threads
  I'm trying to make a simple webcrawler that gets links and images off of websites. I maintain a list of links to visit, and in my run() method I take one of the links, and send it to a processing thread I have and call start on the thread:run() {while(true) {Thread t = new Thread ( new Processor(url) );t.start();}}However, the program runs really slowly, and I was wondering if there was somethign I could do t ospeed up the threads.   
  1. Don't create too many threads. 20-50 threads at once are maybe enough.2. If you write a crawler, have it obey to robots.txt AND don't hammer the site, ie. have a short delay (1 sec) between requests to the same site. If you crawl many sites at once, your crawler will still run at full speed.;  Use a Thread Pool instead of constantly creating new threads.I would also agree with the robots.txt and timing comments above.   ;  Thread pool is included in Java 5 Standard.   ;  I would use a queue and just a single thread.  Each item you pull off the page, add it to the bottom of a queue (or Vector in Java) and have the thread just pull off each item from the top of the queue.  If the queue empties, you can have your thread sleep for a while or something.  Vectors are also nice to use because they're synchronized.Edit: Actually, I mean use 2 threads and a single queue.  One thread pushes items onto the queue, the other pops them off and processes them.   ;  java.nio ?   ;public void run() {  while(true) {    Thread t = new Thread ( new Processor(url) );    t.start();    try { Thread.sleep(500);    catch(InteruptedException e){}  }}Anytime you have a thread that has a short run loop, you should introduce a sleep or it will hog CPU and is very hard for the O/S to break out of to allow other threads CPU time.I think Thread.yield() will work too.   