What is the notation for inserting algorithms in scientific papers
To the best of my knowledge, there is no standard notation for algorithms across computer science fields. However, looking through previous editions of the venue you plan to send your paper to should give you a good impression for how algorithms are typically denoted.

That being said, most papers I know just use a very simple procedural notation, basically pseudo-code in a simplified structural programming language. The reason is likely that such a notation has the least "hidden assumptions" in contrast to what your average computer scientist will know and expect. Contrary, your example seems ripe with symbols and expressions that are certainly not globally understood outside of AI. Using a "compact, expressive" notation loses much of its appeal if you then need to spend half a page to formally define your notation.