This work presents a novel data-driven symmetry search method that can reveal all relevant permutations in molecules and materials present in a given dataset. These permutations are used to differentiate atoms based on their surrounding chemical environments, allowing us to improve the accuracy of MLFFs by adjusting the level of symmetry excercised by the model. We start from a cloud of disconnected vertices that represent atoms and establish connections between them for all representative configurations in a given dataset using the van der Waals radii and interatomic angles. Only the most frequent links are used to build the final graph. In the case of the multicomponent system, the symmetry independence of separate components is analyzed by a coarse-graining procedure, and the final graph is modified accordingly. The final graph serves as an input for the graph automorphism search algorithm. This algorithm produces three primary outputs: (i) the total number of permutational symmetries, (ii) a minimal set of symmetry operations sufficient to generate all permitted permutations of vertices, and (iii) the orbitsequivalence classes of the graph's vertices under the permitted permutations. Additionally, the constructed graph representation facilitates the analysis of local atomic environments, particularly useful in cases where reliance on global symmetries leads to overly complex or inadequate descriptions of the system.