Different capacity lithium polymer batteries in parallel
short answer yes, it is a problem... the weaker battery will start causing damage on the better one. it will cause current flow from one to the other battery and in best case you will suffer early degradation of both battery's.

Edit
combine two packs by two diodes

This is the solution with only one pack at a time

circuit diagram

Thisone I collected some 10-15 years ago from a model plane magazine,
hope you get the idea