Formally describing a new domain specific programming language
There are two parts of the semantics you will need to describe:


static semantics: structure of well formed programs
dynamic semantics: meaning of running programmings


Static semantics usually take the form of type systems. I'd recommend looking at the books by Benjamin C Pierce and Robert Harper. Alternatively, you could write the rules of well-formed programs as a collection of (formal or informal) statements describing the validity conditions of your program. The formal approach taken by type systems provides a well-known framework for such specifications.

There are many formalisms for expressing the dynamic semantics to consider as your basis:  


denotational semantics --- gives meaning  in terms of semantic domains
operational semantics --- gives meaning in terms changes to an abstract machine states
axiomatic semantics --- gives meaning in terms of assertions about the state of the machine
algebraic semantics --- semantics is defined in terms of algebraic laws.


There should be plenty of books out there on these topics. A good broad introductory one is 
Nielson and Nielson's Semantics with Applications. A downloadable version is available from the authors' site.
There is also a newer edition, but I haven't seen it.

Within each of these, there are choices to make. For example, if you decide to use operational semantics, there are large-step semantics, small step semantics, semantics based on labelled transition systems, semantics based on rewriting logic.


In Composing contracts: an adventure in financial engineering, 
Simon Peyton Jones et al gives denotational semantics to a financial DSL.
The Denotational and Static Semantics of a Domain-Speciﬁc
Language for Flow-Network Design by 
Assaf Kfoury presents a denotational semantics for a flow network DSL 
Andrei Lapets' thesis Algebraic Semantics of Domain-Specific Languages
discusses algebraic semantics of DSLs. 
There are many more: this, that, one in terms of abstract machines, another one.


What kind of approach you should take will depend on what kind of language your DSL is, what kind of experience you have with designing semantics, and your personal preference.

Without any concrete details, I can help no further.