What could be the source of sediment found in washing machine?
Kids. Pockets. Dirt. 

Even adults trek in a lot of dirt on their clothes. I would worry about it. That's what the filter is for and now you know how to clean it.