High Power Series Pass Regulator Design
Before you go any further, calculate how much power the transistor needs to dissipate.  The answer will probably tell you to find a better design.  In this case, efficiency isn't so much about wasting power, but having to deal with the heat from the wasted power.  You can buy a lot of electronic parts for the cost of having to remove 100 W of heat from a transistor.

Even with your existing proposal, put the cap after the pass element, not before it.  That way you aren't always starting with the absolute worst input voltage.  The average will be lower.  With the cap immediately after the rectifier, you're always dropping 170 V to 90 V with a linear regulator.  Even at the steady current of 1 A that 80 W, and 250 W for a few seconds at the end!

If you want something simple, find a transformer that drops the incoming AC voltage so that you get the desired voltage right out of the full wave bridge followed by a large cap.  Experiment with a variac to see what you need.  Even just buying a variac and dedicating it to this purpose will be cheaper than keeping a transistor that dissipates over 100 W from blowing itself up.  You might be able to find some old surplus auto-transformer with enough fixed taps that one of them gets you what you want.

I just checked, and you can get a brand new 320 VA variac for $65.  That's probably cheaper and certainly easier than a massive heat sinking and cooling project.

Again, you need a better design, not some magic answer how to make your current scheme work.

Added

You now say you have a limited volume of 10" x 5" x 3".  In that case, you can't use your original scheme.  That's not enough space to get rid of the large heat it would produce while still keeping the components cool enough to not fry themselves.

A transformer capable of stepping down the voltage so you can rectify it and use it directly wouldn't take more volume than you have.  However, finding one off the shelf that fits your geometry is unlikely.

What you need is a proper switching power supply.  If power factor isn't a concern, then you just full-wave rectify the AC into a cap as you intended before.  However, now you PWM that to drive the motor.  Doing this at 170 V and 300 W is not a beginner project, but it certainly can be done.  It would fit into your space with room to spare.