New bike's disc brake rubbing the pads
Yes, the body of the brake that holds the pistons is called the caliper.

The thin metal is a spring that keeps the pads help up against the pistons. It should be there.

First, make sure the caliper is actually aligned properly. Try looking through the caliper to see if the pads are touching the rotor. A bright light and a sheet of white paper held behind the caliper can help you see gaps between pad and rotor. Have a look at the Park Tool video below for how to align the caliper.

If both pads are touching the rotor the caliper the pistons may have been pushed out a little too far. This happens if the lever is pulled when the wheel is out of the bike. Take the pads out and push the pistons back in using a flat head screwdriver or tire lever.

Another possibility is that the pads are so new they simply need wearing down a little. You will need to break the pads in in any case. Here's a good procedure for doing that: https://www.bikeradar.com/us/gear/article/how-to-bed-in-disc-brake-pads-31337/. Break the pads in then try re-aligning the caliper.