Finding number of possible arrangements of books on a shelf with a Monte-Carlo approach
I don't really understand your question. So I'll take things to the extreme and assume there are n1 books, n2 categories, n3 super categories... etc.
This whole hierarchy can be represented as a rooted tree, and the number of valid permutations would be the number of postorder traversals of the tree (or equivalently, preorder).
(I don't expect this will answer your question, I just thought it was interesting)
Also, if you'd like to find some middle ground between your specific example and my extremely overgeneralised... whatever this is... it might be easier to come up with a reasonable answer.
(edit...)
Some more points
This is not specific to matlab, just DSA in general
I still maintain that randomness is not the way to go for this problem.
As far as data structures go using arrays is fine, but for generality/scalability I suggest encoding the categories into the lists as well.
tree = {{1,2,3},{1,2},{1,2,3,4}} &lt;- a cell array lets you structure data
Also, it's inefficient to check each new permutation against every saved entry. The usual thing to do is to use a set (a hashmap or a hashtable). But this adds the problem of computing a hash of the array (or finding some succinct unique id)