How to write a custom editor for a new programming language?
There has been a shift during the last 5-15 years in how editors and IDEs are built. It was recognized that IDEs have to perform a lot of the same tasks that compilers do.

In fact, in order to support things like lexical highlighting, syntax highlighting, semantic highlighting, refactoring, warnings ("yellow squigglies"), errors ("red squigglies"), intelligent code completion, and all the other things that we have come to expect from a modern development environment, the IDE pretty much is a compiler. It needs to do almost all the things, except for optimization and the actual code generation. But, lexing, parsing, semantic analysis, type inference, type checking, symbol resolution, overload resolution, implicit resolution, control flow analysis, data flow analysis, etc., they all need to be performed. Modern IDEs are actually even expected to do more work than the compiler does, such as NULL checking and tracking for C-like languages, finding unreachable code or unused variables and parameters, finding redundant checks, etc. (For example, I recently found a bug in some of my JavaScript code by the editor telling me based on type information in the documentation comments that the then branch of an if statement was unreachable because the condition I was checking for was impossible.)

So, the change that I alluded to above, is that editors and IDEs are starting to use the actual compiler to perform all this analysis. This leads to some interesting requirements for the compilers, because the way they are used in IDEs and the way they are used as batch compilers is fairly different:


Batch compilers deal with complete programs, IDEs deal with incomplete programs. (Analysis while you type.)
Batch compilers deal with working programs, IDEs deal with broken programs. (Most of the time, while you write the program, it will not be in a working state.)
Batch compilers can take as much time and as much memory as they want, IDEs must ideally give feedback within milliseconds after each keypress, and the "IDE compiler" must share its memory with the editor and the IDE.
A batch compiler's job ends as soon as it hits an error, an IDE only gets started. (E.g. giving hints about how to fix the error, or analyzing the rest of the file.)


This pretty much requires a high degree of re-entrancy, asynchrony, concurrency, and parallelism in the compiler. It requires it to be built in a modular design so that you can separate the optimizer and code generator. It requires it to be built as a library, not just a command line tool. It requires it to be built with open access to the parse tree, AST, flow graphs, etc.

So, not only are modern editors and IDEs built to take advantage of compilers, but modern compilers are built to be embeddable into IDEs.

One of the most high-profile compilers that were explicitly built to be able to be embedded into IDEs is Microsoft's Roslyn compiler for C# and VisualBasic.NET. (Actually, it was built to be even more than that: it was built to be a general platform for language analysis of and language tooling around C# and VB.NET that is easy enough to use that non-compiler writers can write e.g. their own static analysis rules.) Anything you see in Visual Studio, from IntelliSense to highlighting to squigglies to hints to errors to warnings … is built on top of Roslyn.

One of the reasons for creating Clang was that most existing C compilers were not designed to be used as anything but a command line batch compiler. Clang is explicitly designed to be embedded into IDEs and editors. As the documentation says, Clang is designed to Support Diverse Clients.

The Scala Compiler has long had a mode called the Presentation Compiler, which is an asynchronous version of the first 4 (of 27) phases of the compiler. The new compiler for Scala 3 is built from the ground up to support IDEs.

Going further, Microsoft has defined the Language Server Protocol originally for the TypeScript plugin in Visual Studio Code. The LSP is a JSON-RPC protocol that allows a language analyzer and an IDE to communicate with each other using a standard protocol. This turns the m×n problem of providing high-quality plugins for m languages for each of a set of n IDEs into an m+n problem of providing language servers for m languages and LSP clients for n IDEs.

If you implement a language server for your language, it will instantly work with any editor or IDE for which an LSP client is available (e.g. Eclipse, Atom, Sublime Text, Brackets, Emacs, Vim/Neovim, Visual Studio, Visual Studio Code, IDEA, …) And when you implement an LSP client for your IDE or editor, it will instantly work with any language for which a language server is available (e.g. Idris, C, D, Go, ECMAScript, TypeScript, ActionScript, Lua, PHP, Python, Ruby, Haskell, Erlang, Haxe, Elm, Elixir, even some not-so-obvious ones such as IBM Enterprise COBOL for z/OS maintained by IBM itself).

So, coming back to your question


  Let's say I've finished my compiler for my new language. Next step is write a custom editor with syntax highlighting (type checking and semantic can wait), is there a cookbook or any kind of guidance to accomplish this project?


The first piece of guidance on how to build a custom editor with syntax highlighting is to not build a custom editor with syntax highlighting. Build a language server.

The second piece of guidance is to not finish your compiler and then start working on the editing experience. Instead, build the compiler from the ground up to support the editing experience.