Is there any objective downside to riding a women's bike as a man?
If the bike fits, ride it.  

The average woman has different dimensions to the average man, so the make bikes with different dimensions to better suit the avenge woman and call it an womans bike (and, as you have noticed, they paint them a different co lour). Problem with all these averages is very few people (if any) are average and the range of sizes of men is significantly more than the difference in size of the average man and average woman.  

Probably the biggest difference is an easy one to change, the saddle. You may want to go for a narrower saddle than the bike comes with.