How does AVRDUDE communicate with the programmer?
This is more belongs to stackoverflow and this is more related to the device drivers. However,

Nowadays USB device drivers are written in ring3 instead of ring0. In English they are user mode drivers. Any modern operating system exposes interface level functionality of USB stack to the user mode by API. And there are libraries like libusb , winusb to access them in user mode, so in theory, you only need to study how to use a library like libusb. 

So the AVR dude, with enough permissions , it can access it's endpoints and write and read from them. So that's how it works.

And from the avrdude source, this is the file which defines the how to communicate with my programmer, ( sorry I'm using usbasp ).
http://svn.savannah.nongnu.org/viewvc/trunk/avrdude/usbasp.c?root=avrdude&amp;view=markup

You can see that it's using libusb if on Linux. 

void usbasp_initpgm(PROGRAMMER * pgm)
{
  strcpy(pgm-&gt;type, "usbasp");

  /*
   * mandatory functions
   */

  pgm-&gt;initialize     = usbasp_initialize;
  pgm-&gt;display        = usbasp_display;
  pgm-&gt;enable         = usbasp_enable;
  pgm-&gt;disable        = usbasp_disable;
  pgm-&gt;program_enable = usbasp_spi_program_enable;
  pgm-&gt;chip_erase     = usbasp_spi_chip_erase;
  pgm-&gt;cmd            = usbasp_spi_cmd;
  pgm-&gt;open           = usbasp_open;
  pgm-&gt;close          = usbasp_close;
  pgm-&gt;read_byte      = avr_read_byte_default;
  pgm-&gt;write_byte     = avr_write_byte_default;

  /*
   * optional functions
   */

  pgm-&gt;paged_write    = usbasp_spi_paged_write;
  pgm-&gt;paged_load     = usbasp_spi_paged_load;
  pgm-&gt;setup          = usbasp_setup;
  pgm-&gt;teardown       = usbasp_teardown;
  pgm-&gt;set_sck_period = usbasp_spi_set_sck_period;

}


You can see that each programmer supported by avrdude have to support those functions, so you can understand this abstraction and how the usbasp.c file convert them into programmer specific. I encourage you to read the source code of your programmer. Good luck.

FYI:
For your programmer usbtiny, the source is here: