time. Spain also responded with no small amount of privateering, using the recaptured city of Dunkirk as a base for its Dunkirk Raiders to molest Dutch, English and French trade. More seriously, the Portuguese part of the empire, with its chronically undermanned African and Asian forts, proved nearly impossible to defend adequately, and with Spain so fully engaged on so many fronts, it could spare little for their defense. Spain also had to deal with Ottoman backed Barbary piracy in the Mediterranean - a vastly greater menace than Caribbean piracy, as well as Oriental and Dutch piracy in the waters around the Philippines. The growth of Spain's empire in the New World was accomplished from Seville, without the close direction of the leadership in Madrid. Charles I and Philip II were primarily concerned with their duties in Europe, and thus control of the Americas was handled by viceroys and colonial administrators who operated with virtual autonomy. The Habsburg kings regarded their colonies as feudal associations rather than integral parts of Spain. No Spanish king could, or did, visit the colonies, either. The Habsburgs, whose family had traditionally ruled over diverse, noncontiguous domains and had been forced to devolve autonomy to local administrators, replicated those feudal policies in Spain, particularly in the Basque country and Aragon. This meant that taxes, infrastructure improvement, and internal trade policy were defined independently by each territory, leading to many internal customs barriers and tolls, and conflicting policies even within the Habsburg domains. Charles I and Philip II had been able to master the various courts through their impressive political energy, but the much weaker Philip III and IV allowed it to decay, and Charles II was incapable of controlling anything at all. The development of Spain itself was hampered by the fact that Charles I and Philip II spent most of their time abroad; for most of the 16th century, Spain was administrated from Brussels and Antwerp, and it was only during the Dutch Revolt that Philip returned to Spain, where he spent most of his time in the seclusion of the monastic palace of El Escorial. The empire, held together by a determined king keeping the bureaucracy together, experienced a setback when a less-trusting ruler came to the throne. Philip II distrusted the nobility and discouraged any independent initiative among them. While writers of the time offered novel solutions to Spain's problems such as