Do we create a programming language by writing a compiler? Say, I wanted to create my own programming language. Assuming that I have made all of the decisions about how I want it to look and act, do I just need to write a compiler for it? 

For example, is the high-level Java code anything other than just text and this text is in the correct format for the compiler to accept it and turn it into something else?

My question is, is the creation of a programming language done through a compiler? High-level is fine.
The short answer is no. 

You can think of a programming language as a mathematical formalism used for expressing computation. A compiler/interpreter is a just a piece of an actual software that carries out that computation, and should not serve as the language specification.

That being said, in addition to lexical and syntax specification of a language, you should also define semantic specification, i.e., what does a (syntactically correct) program written in your language actually mean. The obvious semantics to start with is operational semantics, where the meaning of a program is given in terms of how the program actually runs. That is, it (mathematically) precisely defines how programs execute. On top of this semantics you should build an actual compiler/interpreter with optimizations and so forth. 

Specifying full operational semantics of your language will give you a formal language documentation and it will make you understand your language to the very details. Also, it will allow you to formally reason about some aspects of the language. Writing down operational semantics is really a good habit.

There are other useful semantics as well, such as axiomatic, denotational, and game semantics. However, they are more advanced and typically don't make their way into a compiler/interpreter.