Am I not fit for grad school if I have a bad GRE score? Should I give up?
As some answers for a similar question have pointed out regarding GRE scores, the answer could be maybe. Language too, is not necessarily a hard barrier to doing a PhD in the US, as long as the potential advisor can overlook it.

However, I would strongly urge you to find good reference letters as having both poor letters and poor scores is generally a pretty big red flag, or at the least significantly reduces the competitiveness of your application.

You may still have a chance by contacting faculty directly in less popular or lower ranked schools, which may have a shortage of applicants. This does vary on the field.