title: idea/direction to build a crawler package for meteor
body: <p>I'm thinking of building a package that make building node crawler easy. I've built crawlers before using tools such as webpack and webdriverjs and all the routing and communication back and forth between bot and server is such a pain to setup every time.</p>
  
  <p>Meteor's way seems much sweeter: no explicit routing, client can just invoke functions and it just works. So I intend to build a helper package that you can depend on to build your own crawler package where you only include a bot.js file, within it you emit events and terminate the bot when you're done like this:</p>
  
  <pre><code>var contents=$('.content-to-scrape').html() //getting some contents
  Crawler.emit('contents', contents);
  Crawler.terminate();
  </code></pre>
  
  <p>And on server side you just have to invoke it like this:</p>
  
  <pre><code>Crawler.start('http://somesite.com');
  Crawler.on('contents', callback1);
  Crawler.on('terminate', callback2);
  </code></pre>
  
  <p>I'm having trouble wrapping my head around how to build packages to serve browser bot though. Meteor build system doesn't seem to build for anything other than client and server, and it seems quite complex. So I decided to use webpack for packaging a bundle to serve to browser bot. And I guess I could use <code>WebApp.connectHandlers</code> to add any routing I need. What I'm wondering though is where to put the bundle file created with webpack such that I could serve it using the connect handler.</p>
  
  <p>Tell me if this is feasible with Meteor or whether I'm headed in the right direction?</p>