How to mimic soft-power button in inexpensive toys & flashlights?
I have a lengthy write-up about saving power at Power saving techniques for microprocessors.

One of the sample sketches on that page (Sketch J) does what you ask:

#include &lt;avr/sleep.h&gt;

const byte LED = 9;

void wake ()
{
  // cancel sleep as a precaution
  sleep_disable();
  // precaution while we are doing other things
  detachInterrupt (0);
}  // end of wake

void setup () 
  {
  digitalWrite (2, HIGH);  // enable pull-up
  }  // end of setup

void loop () 
{

  pinMode (LED, OUTPUT);
  digitalWrite (LED, HIGH);
  delay (50);
  digitalWrite (LED, LOW);
  delay (50);
  pinMode (LED, INPUT);

  // disable ADC
  ADCSRA = 0;  

  set_sleep_mode (SLEEP_MODE_PWR_DOWN);  
  sleep_enable();

  // Do not interrupt before we go to sleep, or the
  // ISR will detach interrupts and we won't wake.
  noInterrupts ();

  // will be called when pin D2 goes low  
  attachInterrupt (0, wake, FALLING);

  // turn off brown-out enable in software
  // BODS must be set to one and BODSE must be set to zero within four clock cycles
  MCUCR = bit (BODS) | bit (BODSE);
  // The BODS bit is automatically cleared after three clock cycles
  MCUCR = bit (BODS); 

  // We are guaranteed that the sleep_cpu call will be done
  // as the processor executes the next instruction after
  // interrupts are turned on.
  interrupts ();  // one cycle
  sleep_cpu ();   // one cycle

  } // end of loop


According to when I tested that, it used 0.116 µA when asleep (116 nA). That is well below the rate at which batteries self-discharge.

Basically that code does what you described. It sleeps, when pin D2 goes low it wakes and flashes an LED.

I should point out though that any "development board" Arduino with a voltage regulator, USB interface, and power LED on it will consume considerably more power because of the current drain they consume.

For low-power applications you need the bare chip (with appropriate supporting circuitry, like decoupling capacitors).