What are the modern alternatives to Backus–Naur form and what are their advantages?
Answer 1: The question is meaningless as written.
You are mixing different kinds of notations here that are intended for different purposes.

BNF and ABNF are concrete notations for writing the abstract concept of a context-free grammar.
&quot;Van Wijngaarden grammar&quot; refers either to an abstract type of grammar a la &quot;context-free grammar&quot;, or to a concrete notation for writing this type of grammar. Van Wijngaarden grammars are strictly more expressive than context-free grammars. In fact, parsing them is undecidable in general, making them more like an esoteric programming language in some respects.
ANTLR is a particular software tool for generating parsers. It has a language for writing grammars that it accepts as input. Like most other parser generators, ANTLR includes specialized features in its grammar language for manipulating the generated parser code. For example, it is possible in an ANTLR input file to directly insert Java code that is to be executed by the generated parser when a certain token is encountered. This sort of feature only makes sense in the context of a parser generator.

Answer 2: It mostly doesn't matter.
If, as you say, &quot;you want to invent a new programing language,&quot; there are two situations where writing down a grammar is relevant:

Writing a parser as part of the compiler implementation. Your choice of how to express the grammar here is dictated by how your parser is implemented. Whether you use an ANTLR/Yacc/Bison-style parser generator, a parser combinator library, etc. will determine what your grammar looks like in the source code.

Writing a language specification. Most programming language specifications are written to be read by humans, not computers. Therefore, anything that is sufficiently clear to a human reader is an acceptable choice of notation. It is common for a language specification to define its own grammar notation, rendering the particular choice of notation largely irrelevant. For example, the &quot;Lexical Structure&quot; chapter of the Haskell 2010 Language Report starts with a &quot;Notational Conventions&quot; section that defines how the grammar will be written.