Python: Access mesh selection history data
Don't use the string key. Instead use a direct reference for the layer:

layer = bm.faces.layers.int.new('layer_name')
bm.faces[0][layer] = 0


There is no boolean layer available.

The selection history works only for single verts/edges/faces.

You can use the bmesh operators similar_faces, similar_edges, similar_verts which will return the selection.

Links to the source: bmo_similar.c, bmesh_operators.h