Do we create a programming language by writing a compiler?
The short answer is no. 

You can think of a programming language as a mathematical formalism used for expressing computation. A compiler/interpreter is a just a piece of an actual software that carries out that computation, and should not serve as the language specification.

That being said, in addition to lexical and syntax specification of a language, you should also define semantic specification, i.e., what does a (syntactically correct) program written in your language actually mean. The obvious semantics to start with is operational semantics, where the meaning of a program is given in terms of how the program actually runs. That is, it (mathematically) precisely defines how programs execute. On top of this semantics you should build an actual compiler/interpreter with optimizations and so forth. 

Specifying full operational semantics of your language will give you a formal language documentation and it will make you understand your language to the very details. Also, it will allow you to formally reason about some aspects of the language. Writing down operational semantics is really a good habit.

There are other useful semantics as well, such as axiomatic, denotational, and game semantics. However, they are more advanced and typically don't make their way into a compiler/interpreter.