Selecting a subtree in an array representation of a binary tree
If you change the layout to following:


1 is the root
The left child of i is L(i) = 2*i.
The right child of i is R(i) = 2*i+1.


then formula is simple: binary index of the node is prefix of binary index of any its descendant, f.e. node with index 0b101 is ancestor of any node with index 0b101...

In order to check this property, you need to find index of highest set bit in both numbers hsbit(x) and perform the check descendant &gt;&gt; (hsbit(descendant) - hsbit(ancestor)) == ancestor.

Moreover, it seems that your layout is just the same as mine, but with all indexes decremented by 1.