Is there any existing algorithm or formula that can calculate the complexity of a dataset?
well ... "if n elements each element is composed of m sub elements" sound like you have a set of sets from discrete objects. If this is the case you can go through Combinatorial Analysis like Graph Theory. You can model it in two ways:

1) Using Hypergraphs: You have your elements as hyper-edges of a Hypergraph. This model lets you compute distances, diversities, sizes, and many more graph theoretical measures e.g. look at this, this and this.

2) Simple Graphs: You can create simple complete graphs from each sub-element and connect them to other complete graph (forming cliques) of other sub-elements. See the example bellow:

element1 = [[a,b,c],[b,c,d],[e,f,g]]  
Graph = [(a,b),(b,c),(a,c),(b,c),(c,d),(b,d),(e,f),(e,g),(f,g)]


using this you will have a huge graph (network) of your object on which you can calculate tones of structural and statistical measures.

I assume my thought were naive as your question is not clear and you need to provide more detailed example to get a more accurate answer.

Good Luck!