My department is not concerned with publishing, is this a valid reason to move to a different one?
I did start working there 2 months ago.


and 


  It will not be easy to do that and I will have wasted a lot of time by the time I move out


don't fit well together. If you're worried about the 2 months you've spent, I'd say that's insignificant. If you're worried that it might take (with application processing etc) upto a year, I'd say the same thing. Think of a 30-40 year long research career. A year (or 2 months) is not a long time. 

It sounds like you've done your homework and have a plan for what you want to do, and you're just worried about taking the leap. Take it ! You only get one career, and taking action to further it is always "worth the hassle".