How can I extend an old ceiling box to fit a SmartHome relay?
Add hickeys until you can hang a box like this from it. Grind off the tabs if you don't like them, paint it to match the ceiling. You will probably need a bushing to make the connection from the last hickey to the new box. 



Having pulled the wires through the hickeys, install the relay. Add a cross bar to hang more hickeys, from which you hang the light. Without a 'gap' in the hickeys, you've nowhere to put the relay.



Take the relay with you to the store, find a box and add a hickey; check if it will still fit a cross bar with another hickey sticking into it, with the relay inside, and still have room for wires coming from both hickeys.

I would endeavor to use that company's switches instead of the relay you can't fit, however I am not familiar with them or if they would fit your setup.

The relay doesn't need to be in that box; it just needs the wires... somehow.