What are the ways I can make my circuit communicate with a computer?
There are quite a few different ways you could achieve this. Here are a few:


You could program a small microcontroller like a Microchip PIC16F84A to watch the button and the LED. Whenever the button is pressed, the PIC16F84A sends a character through a FTDI serial chip to a USB port on your computer. On your computer, a program written in Processing watches the USB port and updates the pictures on the screen. Total cost: $20 for some chips and a breadboard.
You could buy a Bluetooth Arduino with a prototyping shield. The Arduino talks over a Bluetooth chip to a Bluetooth card in your PC. A program written in Python listens to the Bluetooth card and draws the appropriate picture on the screen with the Pygame library. Total cost: $150 for a Bluetooth Arduino.
You could attach a webcam to your PC that watches your LED and the button. A program written in Ruby is analyzing every image coming from the webcam, comparing it to the previous image. When it detects a change in the region of the image near the button, it looks at the color of the pixels in the LED region and updates a value in a database. A webpage auto-refreshes using the jQuery timer plugin, and updates an image on the screen based on the value in the database. Total cost: $25 for a crappy webcam.
You could find an old PC with a parallel port and install Ubuntu Linux on it. You connect the LED and the button to pins on the parallel port and then write a program in C that reads address 0x378. Based on the data returned, it makes calls into a C graphics library that does the appropriate screen drawing. Total cost: free, if you can find a PC old enough.
You replace the power button on your computer with the button in question, and the power LED with the LED in question. You replace your BIOS with a version of OpenBIOS that has been modified to display a green circle on the screen, and never boot any operating system. Total cost: probably the cost of a new PC.


But in all seriousness

I think I'd recommend an Arduino and Processing. They're simple and great for beginners. Total cost: $30 for the Arduino.

Best of luck.