best way to log large amounts of data
ELK Stack is a good choice, even though i use elastic search + gray log for logging purposes. Experience with elastic search, several hundreds of GB are not an issue, u can in any case, easily increase capacity adding new nodes to it.

Graylag has an API which you could use in any other app