Double that is not acceptable
According to the Laws of Duplicate Bridge, 2008:


  Law 35
  Inadmissible calls
  
  A. A double or redouble not permitted by Law 19. Law 36 applies.
  
  Law 36
  A. Offender's LHO Calls before Rectification
  If offender's LHO calls before rectification of an inadmissible double or redouble the inadmissible call and all subsequent calls are cancelled. The auction reverts to the player whose turn it is to call [IHO  in this case] and proceeds as though there had been no irregularity. The lead restrictions of Rule 26 do not apply.
  
  B. Offender's LHO Does Not call before Rectification
  When A above does not apply:
    1. any double or redouble not permitted by Rule 19 is canceled.
    2. the offender must substitute a legal call, the auction continues and the offender's partner must pass whenever it is his turn to call.
    3. Law 23 [] may apply. The lead restrictions in Law 26 may apply.
    4. If the call is out of turn .... [not applicable in this instance]


From your description it seems that case B applies: 


You must substitute a legal call;
your partner is required to Pass for the remainder of the auction;  
Director may award an adjusted score at his discretion according to the guidelines of Law 23; and  
if the opponents win the auction there will be lead penalties at your partner's first opportunity to lead, including if he is on Opening Lead. 


The lead penalty in this case will be under 


  Law 26
  B. Other Withdrawn Calls
  
  For other withdrawn calls, declarer may prohibit offender's partner from leading any one suit at his first turn to lead, including the opening lead; such prohibition to continue for as long as offender's partner retains the lead.


Morale: Pay more attention to all those multi-coloured cards on the table in front of you before pulling one out of the box.

Update:

When I was a practicing director, many years ago, it was ACBL Policy that a Director was required to carry a rule book on his person whenever called to the table, and that any player present at the table where a ruling was being made could ask for the applicable Law to be read aloud. I cannot find this in the Rules themselves, and no longer have a current copy of the ACBL Guide for Directors, but I would expect that any polite request to that effect would be accepted by a Director. 

If you believe a Director may be erring in his ruling at the table this is a possible defence; however I would caution against using this wholesale. Directors do not have infinite patience, and are there to make rulings not to teach the Laws. Spend a few bucks; buy yourself a copy of the rules; perhaps even take the Director's Exam to ensure you have a good grasp of the rules. It will help you become a better player and partner in the long run.