What is the purpose of Galileo (or other unremovable chips)
Edit: As Chris points out, this answer is more about the simple AVR Arduinos. For comment on the Galileo and other more complex MCUs, please refer to Chris' answer. 

For Arduino compatibility you can program the Arduino boot loader into the microcontroller using the ICSP pins. You do not need a chip from an Arduino board, you just solder a blank one in. Once the boot loader is on there you can use the Arduino IDE to upload sketches via the UART.

It is not necessary to use the Arduino system to write programs. You include the Arduino library files in a normal C++ project in many different IDEs and use the avr toolchain to compile and avrdude to upload.

Update:

Here is an example of a board I just got back from PCBA. (The picture was sent by the PCB company for checking). Initially I made the board using an Arduino Pro Mini 3.3V, SD card breakout board, LSM303DHLC accelerometer breakout from Pololu, and Real Time Clock (RTC) breakout. Once everything was working I drew up a PCB in Eagle CAD, using the core of Arduino Pro Mini board plus the individual SD card, LSM303DLHC and RTC chips. I also added a 3.3V boost converter so I could get a constant 3.3V off 2 AA batteries.



The board has the ICSP pins broken out. Actually, you will find they are broken out on most Arduino boards. Using another Arduino as ISP I can program the Arduino boot loader via the ICSP pins, and then sketches can be uploaded via the UART using the Arduino IDE. In the IDE the board that is selected is "Arduino Pro Mini 3.3V 8MHz".

The Arduino boot loader is a small piece of firmware that runs on startup. It allows you to send a program over the UART which it stores into FLASH memory itself. It is not absolutely necessary. You can program the chip using the ICSP as well. It is really convenient however. Using software like xloader you can upload Arduino programs without the IDE. Really handy when other people are using the board and you don't want to share your code. Alternatively, you can write a script for avrdude to do it.

This project is for a university so I'm leaving it as Arduino compatible. In a commercial engineering environment, I would choose a more appropriate MCU (one with lower power consumption for a battery device) and rewrite the program to be more efficient and not rely on the Arduino and third-party libraries.