How would you model Rust procedural macros? In Rust programming language one can write a compiler extension function that works on abstract syntax tree, effectively modifying source code before it gets converted into machine instructions.
In other words, macro function has signature
 Abstract Syntax Tree -&gt; Abstract Syntax Tree 

Can these be thought of as higher order functions? Usually higher order functions use function composition to produce their output, not source code manipulation.
I wouldn't consider it a higher-order function.  A higher-order function takes a function as input, or yields a function as its output.  In contrast, it sounds to me like your compiler extension is accepting data as input (the abstract syntax tree) and producing data on its output.
I suppose you could consider the AST a specification of a function, but that seems like a stretch to me.
I'm not sure that what name you attach to it matters much, from a computer science perspective.  A rose by any other name, and all that.