What is the standard notation for introducing syntactical formalism in a paper
There is no general standard way. It depends to some extent on what is
customary in the community you are addressing, on the size and
syntactic complexity of the language you are defining, on the amount
of formality required, and on whether the formalism is intended for
practical use or only for theoretical work.

Typically, if you are defining a programming language, You are likely
to decompose the presentation into a lexical and syntactic part. The
lexical part defines elements of your language such as identifiers and
literals (numbers, strings, or other), usually by means of regular expressions. The syntactic part defines the
various constructs of your language, often with a Context-Free grammar. In the case of programming languages, the most common way of defining a context-free syntax is with the Backus-Naur Form (BNF) which is a metalanguage for syntax definition.

I you a creating a shell command for the Unix system, you will use a
presentation that is quite close to your example.

If you are defining an algebraic formalism, you may define a tree
syntax, or possibly a string syntax based on a context-free formalism,
possibly not caring for syntactic ambiguity issues.

It is up to you to be knowledgeable enough to determine what is
appropriate in your specific case. Existing similar literature should
help you.

Two questions you want to ask yourself when choosing a notation are:


what is sufficient to convey my intent with enough precision, given the purpose and use of the language?
Will it be clear for my readers, given the kind of experience and knowledge they are supposed to have.


Your specific example of being the first to introduce XML syntax, is difficult to answer because the is not really a first. The XML formalism was derived from predecessors such as HTML and SGML, and probably other mark-up languages. One is seldom first, and it is usually wise to restart from the shoulders of those who came before. This applies both to what we define, and to the definitional tools. However, the intended community of users may change, and choices that may be best for a very general public may not be so good for computer specialists, or for theoretical work, the converse being also true. On the other hand, people prefer things close to what they already know.

If you want to introduce a syntax near or derived from XML, the above applies, and you should probably use the same definitional tools But you should probably make sure that your variations are necessary. In such a case, you may also rely on what people may already know of XML, which is fairly widely known, and insist on the differences, on the ways your language departs from XML.