I am starting a big data initiative for my startup. In 2018 is there any reason to use Hadoop at all since Spark is touted to be way faster due to it primarily not writing the intermediate data to disk as Hadoop’s MR.

I realize Spark has a higher need for RAM But that would be just one time CAPEX costs that would pay for itself?

In general unless there are legacy projects why should one pick up Hadoop at all since Spark is available?

Would appreciate real world comparisons of the two, gotchas etc.?

Alternately are there use cases that Hadoop can solve but Spark cannot?

—————-comment below for actual problem————

I would use YARN as the resource manager with HDFS as the file system for Spark.
Also realize that as Spark intersects quiet a bit with Hadoop ecosystem. 

Comparos are :

1. Mapreduce vs Spark code
2. SparkSQL vs Hive
3. People mention Pig too but not a whole lot of people want to learn custom querying. And if I had to use Pig as a data scientist why wouldn’t I use say an Apache NiFi with Hadoop?

Also not sure how Spark handles the following:

1. If data does not fit in RAM then what ? Back to a disk based paradigm (not talking of streaming use cases here..) so no better than Mapreduce? How does Tez make MR2 better?
2. Hadoop 3 has support for Erasure coding to reduce data replication. What does Spark do?

Where I am unclear is the plethora of overlapping choices. For e.g. streaming alone has:

1. Spark streaming
2. Apache storm
3. Apache Samza
4. Kafka streams
5. CEP commercial tools.(ORacle CEP, TIBCO etc.)

A lot of them use DAG similar to Spark’s core engine so hard to pick one from the other.

Use case:
=========

1. App sends data to middleware until end of event. Event can end specified on periodicity or due to a business condition being met.
2. Middleware must show real time addition of a value (simplifying) sent by users from their app instances. Accepted that middleware is the floor of the actual sum of values and real value can be higher. **Plan to use Kafka streams here to have a consumer that adds all the inputs with minimal latency** the consumer posts to a cache which is polled by apps to show current additive value.
3. Middleware logs all input
4. After event ends a big data paradigm scans through log data and database records to get accurate count by comparing all dB values and log entries (audit) and compare them to the Kafka shown value. Value calculated by this scheme is the final value.

Design choices:
===============

1. I like Kafka because it decouples application middleware and is low latency high throughput messaging. Streams code is easy to write . *Happy for someone to counter argue using Spark Streams Or Apache Storm or Apache Samza instead?*
2. Application itself is Java code on Tomcat server with REST end points for iOS/ Android clients. Not doing client caching due to explicit liveliness of additive value.