Separate construction phase from constructed phase
First of all, you are right, what you have there is not a builder class, but more like a factory class. Which is roughly equivalent to a static factory method.

Now, the question of using constructors vs static factory methods is one over which there has been a lot of talk, and for the most part no conclusive, universally-applicable answers. (Just search for "c# constructor vs static factory method" and you will see.) However, specifically for your case, where both the constructed class and the code constructing it are private within another class, the advantages of "discoverability" and "recognizability" that the new keyword has are not applicable, so I'd definitely go with a factory method or class.

With regards to whether you should be using a factory class or a static factory method, I think that the factory class is fine. The overhead for the generation of an additional object which gets immediately discarded is negligible, (especially compared to the amount of work that the object will do,) and might even be optimized away by the compiler. On the other hand, having a factory class instead of a static factory method could turn out to be useful if in the future you decide to introduce some state into your builder.