Classifying and counting database entries using Scala map and flatMap
Let me start with out-of-the-box solution I would use if I were and after that we'll discuss your code. I assume df is equivalent to the following structure:

val df = Seq(
  (1, "John", "S1", "green"), (2, "Shaun", "S2", "red"),
  (3, "Shaun", "S2", "green"), (4, "Shaun", "S2", "green"), 
  (5, "John", "S1", "yellow")
).toDF("id", "name", "eid", "color")


All you really need to achieve desired output is pivot:

df.groupBy("name", "eid").pivot("color").count().na.fill(0).show

// +-----+---+-----+---+------+ 
// | name|eid|green|red|yellow|
// +-----+---+-----+---+------+
// |Shaun| S2|    2|  1|     0|
// | John| S1|    1|  0|     1|
// +-----+---+-----+---+------+


About your code:


Don't fetch more data from a DataFrame than you really need. Once you convert DataFrame you don't longer benefit from early projections, selections and other Catalyst optimizations.

It means you have to fetch all the data from external source (like a database) or off-heap storage even if downstream processing requires only a small fraction of it. In other words project early:

df.select($"name", $"eid", $"color").rdd.map { ... }

Avoid dealing with RDD[Row]. It is like a Seq[Any] - nothing you really want in your code. Instead you can use Dataset encoders:

df.select($"name", $"eid", $"color").as[(String, String, String)].rdd.map {
  ... 
}

Never use list concatenation for reduction. Since List.++ is O(N) operation and you apply it in a loop overall complexity is roughly O(N2). If you really want to group data use groupByKey. 
But don't group if operation can be expressed using reduceByKey with a truly reducing (requiring roughly constant memory) function.

There many ways you can approach this for example with aggregateByKey (note that I intentionally use mutable buffer):

import scala.collection.mutable.Map

val pairs = df
  .select($"name", $"eid", $"color")
  .as[(String, String, String)]
  .rdd.map { case (name, eid, color) =&gt; ((name, eid), color) }

def seqOp(acc: Map[String, Long], x: String) = {
  acc(x) = 1L + acc.getOrElse(x, 0L)
  acc
}

def mergeOp(acc1: Map[String, Long], acc2: Map[String, Long]) = {
  acc2.foreach { case (k, v) =&gt; acc1(k) = v + acc1.getOrElse(k, 0L) }
  acc1
}

pairs
  .aggregateByKey(Map.empty[String, Long])(seqOp, mergeOp)
  .map {
    case ((name, eid), vs ) =&gt; 
      (name, eid, vs.get("red"), vs.get("green"), vs.get("yellow"))
  }.toDF("name", "eid", "red", "green", "yellow")