Should I use the Rules module to implement custom code when performing state transition?
Using the Rules module for this is indeed an option, whereas you should be able to implement your "custom code" using the Rules Action to "Execute custom PHP Code". If you do, make sure to not forget the PHP filter (otherwise you'll run into "Rules "Execute custom PHP Code" option not available ...").

Of course, that required 'PHP filter' might be your show-stopper. BUT: are you sure you need "custom code" at all? There may be alternative (site building only) techniques to even not need that "custom code" (and hence you'd avoid the need for that PHP filter also). And/or as a variation, you could write your own (custom) Rules Actions also ...

Not to forget these benefits also:


Less custom code (= easier to maintain and/or upgrade to future Drupal versions).
Possible out-of-the-box integrations with other modules, such as Flag, VBO etc.