Is there such thing as "almond tofu," and if so, is it a misnomer?
It's semantic nitpicking.

Tofu is defined as soy milk, curdled and pressed. Some people who don't care about oriental culinary tradition think of tofu as any non-dairy milk that is curdled and pressed into a curd.

Technically your almond tofu would be almond milk curd or some such.

In reality your coworker is being pedantic and I would accept the term "almond tofu" as a perfectly understandable, non oriental, colloquialism.