Is relativization well-defined?
You are exactly right. The relativization operation $B\mapsto B^A$ is not well defined. 
P and PA are independently defined objects. The names are suggestive, but you cannot formally define PA from the set P. (You can define P from PA by setting A to be the empty set.)

Think of PA as being some kind of generalization of P, which equals P when A is empty, but otherwise may be different. Now if you only know the set P, it is not clear how to generalize this to get PA. As an analogy, if I asked you to generalize the real numbers, it's not clear what generalization I'm looking for. Am I thinking of fields, rings, vector spaces, etc.? The reason this happens is that while P is merely a set of languages, PA is defined in terms of a machine. This machine has the property that when A is empty it decides exactly the same languages as P. You could come up with some other machine, let's call it QA, which also has the property that when A is empty decides the same languages as P. This does not mean that PA = QA for all A. This would be analogous to asserting that if f(0)=g(0), then f and g are the same function.

Perhaps this post by Terence Tao will be helpful.