When should custom-properties be used instead of 'bpy.props'?
Definition

bpy.props need to be defined.  eg scene.myprop is not available to scripts until it is defined via bpy.types.Scene.myprop = bpy.props.BoolProperty()
It is however saved, once set in a file, and the set value can be retrieved once defined again.  

Type Casting

Properties defined using bpy.props are type cast, and expect a value of that type, or are converted 

&gt;&gt;&gt; C.scene.my_prop = 1
&gt;&gt;&gt; C.scene.my_prop
True
&gt;&gt;&gt; C.scene.my_prop = "XXXX"
Traceback (most recent call last):
  File "&lt;blender_console&gt;", line 1, in &lt;module&gt;
TypeError: bpy_struct: item.attr = val: Scene.my_prop expected True/False or 0/1, not str


Custom props 

scene["my_prop"] = 1
scene["my_prop"] = "XXXX"


Scope

bpy.props are defined for all objects of that blender type, for your example scene.my_prop will be available to all scenes, whereas scene["myprop"] only for the scene instance it is defined on.  I choose custom props for more "one off" usage.

Update Method

An update callback method is available to bpy.props properties.  This is a very useful tool, that is not available to custom properties.  Similar to using an "on_change" event handler.

def jangle(self, context):
    print("update called on scene ", self.name)
    # do something based on the value of "my_prop"
    if self.my_prop:
        print("It's on")
    else:
        print("It's off")
    return None # always return None from an update.

bpy.types.Scene.my_prop = bpy.props.BoolProperty(update=jangle)


UI

both can be decorated in the UI, to some extent.  To add a name and description to each.

bpy.types.Scene.my_prop = bpy.props.BoolProperty(name="My Property",
                       description="This is my bpy.props boolean prop")


and for custom props

    rna = {"name": "My Property"
           "description": "This is my custom boolean prop",
           # can also store other data that can be set by script
           # and used as a driver var
           "anyolthing": 13
          }

    scene["my_prop"] = False
    if "_RNA_UI" not in scene.keys(): #  keys() returns a list of id props
        self["_RNA_UI"] = {}

    # define the UI propeties of "my_prop"
    self["_RNA_UI"]["my_prop"] = rna


note the rna_ui dictionary can have self defined property names, as long as the data type is int, float, bool, string.  They will not effect the UI, but are a handy way to store "hidden" data.  The rna dictionary is a handy way to update the min and max and description on a per property basis, to make descriptive and effective visual sliders for instance.

rna Access

Once a custom property is set, the property name will be a member of the, in this case, scene.keys() list, or as name/value pairs in scene.items()

All bpy.props properties  will be available via scene.bl_rna.properties.keys() but only in the scene.keys() list once initialized, ie given a value other than the default.