Is it an antipattern to avoid API Gateway for certain tasks?
You could have a separate API gateway for that kind of traffic.

Check out the "Variation: Backends for frontends" section of the following article:

https://microservices.io/patterns/apigateway.html