Python __new__ and __init__ usage
First, the thing that was most obvious in your code was that you never used a space after a comma.

Second, you have the Matrix class where you override __new__ and __init__, however it would be work just fine if you'd use class Matrix(Dim): pass.

That's where I started to wonder why you define a Dim class and than inherit from it for Matrix and Vec, instead of code Matrix and subclass Vec from it. Any reason?

I would move the tests from Dim.__new__, except for the numbers test, to __init__. The iter test is not needed, you'll get an error from the list constructor or the for loop anyway.

One issue I see is that Matrix('abc') will result in endless recursion, because strings are iterable. Those are the places where I regret the absence of chr type in python :)

In Vec.__new__, I would let others customize the way they inherit from, why do you check if the class is a Vec or a Dim?

Last tip, when inheriting types like list or str, you have to override all the operator overloading methods if you plan on doing things like Vec((0, 1, 2)) * Vec((2, 3, 4))

Also:

[Vec.__name__,Dim.__name__] use tuples here, memory management

test2DMat = [[1,2,3],[4,5,6],[7,8,9]] I would break lines here