Procedures using TVPs are slower when the TVP numeric value gets larger?
Table Variables, even when used as a parameter (TVP), are given very poor cardinality estimates as opposed to Temp Tables which are much more accurately estimated. This difference is especially noticeable as the amount of data grows that's being utilized in a TVP vs Temp Table. If you looked closely at the Estimated Number of Rows vs the Actual Number of Rows in the execution plans of each implementation, you should see Temp Tables are a lot more accurately estimated.
You can read more on TVPs and their downsides in this Jeremiah Peschka post. Specifically the Gotchas section:

First: the table variable that comes in as a table valued parameter
cannot be changed. You’re stuck with whatever values show up. No
inserts, updates, or deletes can be applied.
Second: table valued parameters are still table variables – they get
terrible cardinality estimates.
We can get around both of these problems with the same technique –
copy the contents of the TVP into a temp table.

Additionally, TVPs used in procedures can result in parameter sniffing issues, as this other post details. This quote adds some specifics to the cardinality estimates you'll encounter with TVPs regardless of how big the actual Table Variable is:

Table Variables (unless you Recompile, or use a Trace Flag), will
sport either a 1 or 100 row estimate, depending on which version of
the Cardinality Estimator you use. The old version guesses 1 row, the
new guesses 100 rows.

This is an additional good article on cardinality estimate issues that result from Table Variables, by Pinal Dave.

One key reason why having a bad cardinality estimate such as an under-estimate in this case, is because it'll result in the SQL Engine under-provisioning the necessary server resources to process the query and serve the data. For example, your query is likely requesting much less memory than it needs for it to be processed because of the low cardinality estimate making the SQL Engine think there's a lot less rows going to be returned than there actually will be. The larger the table variable gets, the larger the discrepancy between the estimate vs actual.

You should pretty much always opt for Temp Tables when possible for the very reason they have many more performance benefits than a Table Variable, and can do almost everything a Table Variable can do and more. In the cases where you need to use a Table Variable, selecting it into a Temp Table first and using that Temp Table in your subsequent querying is the way to go.