Is our data "Big Data" (Startup) I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products.

In essence, we will be consuming real-time coordinates from vehicles and users and performing analytics and machine learning on this incoming data. This processing can be rather intensive as we try predict the ETAs of this entities matched to historical data and static paths. 

The approach they want to take is using the latest and most powerful technology stack, that being Hadoop, Storm etc to process these coordinates. Problem is that no-one in the team has implemented such a system and only has had the last month or so to skill up on it.

My belief is that a safer approach would be to use NoSQL storage such as "Azure Table Storage" in an event based system to achieve the same result in less time. To me it's the agile approach, as this is a system that we are familiar with. Then if the demand warrants it, we can look at implementing Hadoop in the future.

I haven't done a significant amount of research in this field, so would appreciate your input.

Questions:


How many tracking entities (sending coordinates every 10 seconds) would warrant Hadoop?
Would it be easy to initially start off with a simpler approach such as "Azure Table Storage" then onto Hadoop at a later point?
If you had to estimate, how long would you say a team of 3 developers would take to implement a basic Hadoop/Storm system?
Is Hadoop necessary to invest from the get go as we will quickly incur major costs?


I know these are vague questions, but I want to make sure we aren't going to invest unnecessary resources with a deadline coming up.
Yes, this is a how-long-is-a-piece-of-string question. I think it's good to beware of over-engineering, while also making sure you engineer for where you think you'll be in a year.

First I'd suggest you distinguish between processing and storage. Storm is a (stream) processing framework; NoSQL databases are a storage paradigm. These are not alternatives. The Hadoop ecosystem has HBase for NoSQL; I suspect Azure has some kind of stream processing story.

The bigger difference in your two alternatives is consuming a cloud provider's ecosystem vs Hadoop. The upside to Azure, or AWS, or GCE, is that these services optimize for integrating with each other, with billing, machine management, etc. The downside is being locked in to the cloud provider; you can't run Azure stuff anywhere but Azure. Hadoop takes more work to integrate since it's really a confederation of sometimes loosely-related projects. You're investing in both a distribution, and a place to run that distribution. But, you get a lot less lock-in, and probably more easy access to talent, and a broader choice of tools.

The Azure road is also a "big data" solution in that it has a lot of the scalability properties you want for big data, and the complexity as well. It does not strike me as an easier route. Do you need to invest in distributed/cloud anything at this scale? given your IoT-themed use case, I believe you will need to soon, if not now, so yes. You're not talking about gigabytes, but many terabytes in just the first year.

I'd give a fresh team 6-12 months to fully productionize something based on either of these platforms. That can certainly be staged as a POC, followed by more elaborate engineering.