Toy Vector Library - Magnitude & Unit Vector Functions
It is custom/industry de facto standard to place the parameter to be changed first, as done in memcpy, strcpy etc (this tradition goes all the way back to assembler).

So you should do void vec2_unit (vec2_t* dst, const vec2_t* src);.
Regarding in-place or not, it is mostly a matter of style in this case. 

Functions/APIs working on "immutable" objects (objects not modified by the function) are often considered better style when that option is available, since that minimizes the chance of caller-side bugs.

The discussion about in-place vs immutable mostly makes sense for larger data types, where taking a copy of the data object is regarded as costly. A struct with 2 double isn't really that heavy to copy(note 1), so you could design this API with an immutable object interface. You could even get away with passing the structs to/from the functions by value, which is otherwise normally frowned upon.

Note: if these functions will reside in the same translation unit as the caller code (which doesn't seem likely here?), the whole discussion about performance is pointless since they will be inlined anyway in that case.
If you go for the immutable version, it could be written as

void vec2_unit (vec2_t* restrict dst, const vec2_t* restrict src);

This tells the compiler and the caller both that these two objects shall not be the same one. That will in turn increase performance, but it will not allow you to pass the same object as both source and destination.




(note 1) We can assume that systems using double are able to copy it in a few instructions. Systems where copying a double would be lots of work, such as small microcontroller systems, shouldn't be using double (or float) in the first place, since they lack a FPU. I would safely assume that any program using floating point is meant to run on a Cortex M3 or bigger.