I'd like to try using a kernel other than the one provided by my distro -- either from somewhere else, or as customized by me. Is this difficult or dangerous?
Building a custom kernel can be time consuming -- mostly in the configuration, since modern computers can do the build in a matter of minutes -- but it is not particularly dangerous if you keep your current, working kernel, and make sure to leave that as an option via your bootloader (see step #6 below). This way, if your new one does not work, you can just reboot the old one.
In the following instructions, paths inside the source tree take the form [src]/whatever, where [src] is the directory you installed the source into, e.g. /usr/src/linux-3.13.3. You probably want to do this stuff su root as the source tree should remain secure in terms of write permissions (it should be owned by root).
While some of the steps are optional, you should read them anyway as they contain information necessary to understanding the rest of the process.
Download and unpack the source tarball.
These are available from kernel.org. The latest ones are listed on the front page, but if you look inside the /pub/ directory, you'll find an archive going all the way back to version 1.0. Unless you have special reason to do otherwise, you are best off just choosing the "Latest Stable". At the time of this writing, this is a 74 MB tar.xz file.
Note that individual distros usually recommend you use one of their source packages instead of the vanilla tree. This contains distro specific patches, which may or may not matter to you. It will also match the kernel include headers used to compile some userspace tools, although they are most likely identical anyway.
Start with a basic configuration [optional].
You don't have to do this -- you can just dive right in and create a configuration from scratch. However, if you've never done that before, expect a lot of trial and error. This also means having to read through most of the options (there are hundreds). A better bet is to use your existing configuration, if available. If you used a distro source package, it probably already contains a [src]/.config file, so you can use that. Otherwise, check for a /proc/config.gz. This is an optional feature added in the 2.6 kernel. If it exists, copy that into the top level of the source tree and gunzip -c config.gz > .config.
If it doesn't exist, it maybe because this option was configured as a module. Try sudo modprobe configs, then check the /proc directory for config.gz again.
The distro config is not very ideal in the sense that it includes almost every possible hardware driver. This does not matter to the kernel's functionality much, since they are modules and most of them will never get used, but it very significantly increases the time required to build. It is also awkward in that it requires an initramfs to contain certain core modules (see step #4 below). However, it is probably a better starting point than the default.
Note that the configuration options shift and change from one kernel version to the next, and when you run one of the make config programs below your .config will first be parsed and updated to match the new version. If the configuration is from a vastly older version, this may lead to strange results, so pay attention when you do the configuration. AFAIK it won't work at all the other way around (using a config from a newer version).
make config is the most basic but probably not to most people's taste. It is a sequence of questions -- a lot of questions -- and if you change your mind you have to start again.
make oldconfig is like make config except, if you already have a .config from a previous version, will skip questions except those pertaining to new options. There can still be a lot of those and most of them will be irrelevant to you so again, I don't recommend it.
make menuconfig is my (and I think most other's) preferred method. It builds and executes a TUI interface (colored menus that will work on a terminal). This requires you have the -dev package for ncurses installed. It is fairly self-explanatory, except for the seach which is accessible via /; the F1 "help" provides an explanation for the current option. There is an alternate version, make nconfig, with a few extra features, wherein F2 "syminfo" is the equivalent of menuconfig's F1.
make xconfig is a full GUI interface. This requires qmake and the -dev package for Qt be installed, as again, it's a program that is compiled and built. If you were not using these previously, that may be a substantial download. The reason I prefer menuconfig to the GUI version is that option hierarchies are presented using successive screens in the former but open accordion-like in the latter.
One of the first things you should (but don't have to) do is to add a "Local version" string (under General Setup). The reason for this is mentioned in #5 below.
An "initramfs" is a compressed filesystem built into the kernel and/or loaded at boot time. Its primary purpose is to include modules that the kernel will need before it can access those in /lib/modules on the root filesystem -- e.g., drivers for the device containing that filesystem. Distros always use these partially because the drivers are mutually incompatible, and so cannot be all built into the kernel. Instead, ones appropriate to the current system are selected from inside the initramfs.
This works well and does not represent any kind of disadvantage, but it is probably an unnecessary complication when building your own kernel.2 The catch is, if you don't use an initramfs, you need to make sure the drivers for your root filesystem (and the device it's on) are built into the kernel. In menuconfig, this is the difference between an M (= module) option and a * (= built-in) option. If you don't get this right, the system will fail early on in the boot process. So, e.g., if you have a SATA harddisk and an ext4 root filesystem, you need drivers for those built-in. [If anyone can think of anything else that's a must-have, leave a comment and I'll incorporate that here].
If you do want to use an initramfs, you'll have to select the appropriate options in General Setup. There's a skeleton guide to creating one built into the kernel in [src]/Documentation/filesystems/ramfs-rootfs-initramfs.txt, but note that the distros don't do this; they use an external gzipped cpio file. However, that doc does contain a discussion of what should go in the initramfs (see "Contents of initramfs").
Build and install the kernel.
The next step is easy. To make the kernel, just run make in the [src] directory. If you are on a multi-core system, you can add -j N to speed things up, where N is the number of cores you want to dedicate + 1. There is no test or check. Once that's done, you can make modules. On a fast box, all this should take < 10 minutes.
If all goes well, make INSTALL_MOD_STRIP=1 modules_install. This will create a directory in /lib/modules matching the version number of the kernel plus the "Local version" string mentioned in step 3, if any. If you did not use a "Local version" string, be careful if you already have a kernel of the same version that you depend upon, because these modules will replace those.3 INSTALL_MOD_STRIP=1 is optional, for the significance see here.
Here, [VERSION] is exactly the same as the name of the directory in /lib/modules created by make modules_install, which will include the "Local version" string, e.g., System.map-3.13.3-mykernel.
Configure the GRUB 2 bootloader.
Unless grub is not on your root filesystem, this should also correspond to the root= directive on the linux line, which indicates your root filesystem (the one with /sbin/init and /lib/modules). The UUID version of that is root=UUID=[the UUID].
You can look at your existing /boot/grub2/grub.cfg for a clue about the device name. Here's a brief guide to such under grub 2. Once you are happy, run grub2-mkconfig -o /boot/grub2/grub.cfg (but back up your current grub.cfg first). You may then want to edit that file and move your entry to the top. It should still contain a listing for your old (running) kernel, and your distro may have a mechanism which duplicated an entry for the new kernel automatically (because it was found in /boot; Fedora does this, hence, using a distinct title with menuentry is a good idea). You can remove that later if all goes well.
You can also just insert the menuentry into grub.cfg directly, but some distros will overwrite this when their kernel is updated (whereas using /etc/grub.d/ will keep it incorporated).
That's it. All you need to do now is reboot. If it doesn't work, try and deduce the problem from the screen output, reboot choosing an old kernel, and go back to step 3 (except use the .config you already have and tweak that). It may be a good idea to make clean (or make mrproper) between attempts but make sure you copy [src]/.config to some backup first, because that will get erased. This helps to ensure that objects used in the build process are not stale.
Regarding kernel headers et. al.
One thing you should likely do is symlink (ln -s -i) /lib/modules/X.X.X/source and /lib/modules/X.X.X/build to the /usr/src directory where the source tree is (keep that). This is necessary so that some userspace tools (and third party driver installers) can access the source for the running kernel.
Leave the ones used by your distro. If you update the whole system regularly, the distro will install new ones periodically anyway, so this is the "least hassle" option.
Since they are backward compatible (meaning "a program built against a C library using older kernel headers should run on a newer kernel"), you don't have to get too fussy about this. The only potential issue would be if you build a custom kernel and keep it for a while, during which time the distro updates the "kernel-headers" package to a newer version than used to build your kernel, and there turns out to be some incompatibility (which would only apply to software subsequently compiled from source).
[src]/README includes a brief guide to building and installing.
The [src]/Documentation directory contains a lot of information that may be helpful in configuration.
Much of Greg K-H's book Linux Kernel in a Nutshell (available there for free as a series of PDF's) revolves around building the kernel.
Grub 2 has an online manual.
1. "Vanilla" refers to the original, unadulterated official source as found at kernel.org. Most distros take this vanilla source and add some minor customizations.
2. Note that there are circumstances that require an initramfs because some userspace is needed in order to mount the root filesystem -- for example, if it is encrypted, or spread across a complex RAID array.
3. It won't remove modules that are already there if you didn't build them, however, which means you can add a module later by simply modifying your configuration and running make modules_install again. Note that building some modules may require changes to the kernel itself, in which case you also have to replace the kernel. You'll be able to tell when you try to use modprobe to insert the module.
4. This file may be named something different if you used a non-standard compression option. I'm not sure what all the possibilities are.
How do I upgrade the Debian Wheezy kernel offline?
How to create a kernel source tree?
Can I upgrade my Linux kernel from 2.6.32 to 3.0.7?
How to test new kernel releases ?
Where does the Linux kernel reside in memory ?
Which default .config to start from when compiling kernel?