How would you model Rust procedural macros?
I wouldn't consider it a higher-order function.  A higher-order function takes a function as input, or yields a function as its output.  In contrast, it sounds to me like your compiler extension is accepting data as input (the abstract syntax tree) and producing data on its output.
I suppose you could consider the AST a specification of a function, but that seems like a stretch to me.
I'm not sure that what name you attach to it matters much, from a computer science perspective.  A rose by any other name, and all that.