Apache Spark alternatives for local compute I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is &lt; 500,000 items with dimensionality 5 (2 dimensions are free form text).

From what I've started with on Spark, it is heavily geared toward distributed computing and other production related concerns but it has a nice ml library. Is it worth using Spark if I only plan to run my project on a local dev machine level or is there a more appropriate library out there for me in Scala? Something like scikit-learn, but in Scala.
You're right, Spark is intended to scale in a distributed computing environment, but it is absolutely performs well locally.

When running locally Spark will still 'distribute' processing across local Executors, you have options to control the number of CPU cores the Spark job can use, and how many cores each Executor can use. You can get a lot out of using Spark locally, and Spark makes it very easy to scale up to a cluster of machines (e.g. on AWS or Google Cloud)  when your single machine can't handle the tasks and it becomes necessary.

I'd also check out SMILE as an alternative to Spark MLlib in Scala.