Apple dtracehelper file
DTrace in general is a dynamic tracing framework that allows 
users to get information about the running system, including the user-space applications and the kernel.

The /dev/dtracehelper file is used to provider user-defined probes to dtrace. Essentially this means that an application can inform the dtrace system of events happening inside the application to provide high-level tracing capabilities for the user.

I.e. instead of generic events like system calls or signals, it is possible for software such as for example a database engine to provide dtrace with information such as "SQL query started" or "Query plan computed" or what ever the application developer wants.

It is not "used" as such by ordinary users - you use it through the software you installed that then makes information available in dtrace through the helper device.

If you are a developer, I would suggest reading the following documentation on how to add application-specific probes to your program:

http://dtrace.org/guide/chp-usdt.html#chp-usdt

If you really want to manually access /dev/dtracehelper, I suggest looking at the source:

https://github.com/apple/darwin-xnu/blob/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/dtrace.h

(line 1488 and onwards, also take a look at the corresponding .c file)

In particular you would want to initialize data structures such as dof_ioctl_data with dof_helper substructures, and use ioctl() with DTRACEHIOC_ADDDOF and others.

You can find an example of doing this here:

https://github.com/trentm/hello-json-logging/blob/master/node_modules/restify/node_modules/dtrace-provider/libusdt/usdt_dof_file.c