Altium "Where Used" search in multiple projects
I've used a number of ECAD systems over the years, but never Altium specifically. If there isn't a built-in way to get what you want, here's another approach.

I assume you can get a BOM for each assembly in tabular form (e.g., a spreadsheet or equivalent). You just need to add a column to the BOM that contains the assembly ID on every line. Then mash all of the BOMs together into one big table and sort on the part ID column.