Is it possible to do good research on a certain area through self-learning the state-of-the-art of the area?
At some point in a research career you will need to learn directly from papers. Even as a practical programmer, when I wanted the absolute state of the art in an area, I would read academic papers, not wait for the material to show up in textbooks. A researcher often ends up knowing more about their topic than anyone else in the world, not just in their university, and must learn by means that don't depend on access to more knowledgeable people.

That does not mean your proposed course is a wise one at this stage in your career.

As I see it, student research has three objectives:


Learn how to do research
Collect evidence you can do research, in the form of degrees and letters of recommendation.
Contribute to the collective body of knowledge.


You need to decide the weight of these objectives for yourself, but generally the earlier you are in your studies and career, the greater the importance of the first two objectives.

The risk you are potentially taking is sacrificing the first two objectives at a relatively early stage. If your professors do not understand the subject of your research, they may be less able to guide you. It may also be harder for them to judge the quality of your research, which they need to do in order to grant degrees and write recommendation letters.

If you already have really good research skills, the first objective is unnecessary for you. However, if you had that much research experience I doubt you would be asking this question.

You might consider a compromise: Pick some research direction that fits with the research at your university but moves in the direction you want. For example, do a theoretical modeling project that models something your professors do know about.