How to compile a custom kernel

Compiling your own kernel comes with many advantages.
Firstly, it allows you to use the upstream sources directly, so that you do not have to wait for the distribution maintainers to publish new packages.
Secondly, it allows you to configure the kernel however you want. Often your own builds will be a lot smaller than the distribution packages, as distribution maintainers try to squeeze in support for as many hardware configurations as they can. This is a good thing, but it also results in a lot of code that your system isn't using. Custom kernels can get rid of this issue by making the kernel tailored for your system.
Thirdly, you will gain access to the kernel mainline, which are effectively pre-release kernels used for testing purposes. Distributions do not release these. Manjaro would be the only exception that I know of anyway. For better or worse, they offer mainline releases to their users...
Finally, going custom allows you to exert greater control over when systems shall be rebooted into a new kernel. This is important for example since my Raspberry Pi with (at the time of writing) 140 days of uptime, apparently removed the modules for its current running kernel during a distribution update. This means that modules such as FUSE cannot be loaded in that session anymore, unless I reboot into a new kernel. A custom kernel ensures that the package manager won't present such nasty surprises anymore. On servers this is an important factor as well, since often the only reboots there are scheduled ones and don't occur very often.

I should also point out one last thing that people often don't consider: architectures. Lots of distributions only support a handful of architectures, usually x86_64 and aarch64. However, the GCC supports a whole list of architectures, even less common ones such as MIPS and PowerPC. This applies to being able to compile from source in general rather than kernels in particular, but needless to say, having access to every architecture that the GCC supports can be advantageous.

For this tutorial, we'll focus on Arch Linux. For other distributions, it may work, or it may not. In the future I might cover other distributions such as Ubuntu as well.

First off, ensure that you have the appropriate packages installed on your system.
sudo pacman -S base-devel bc jq curl wget gnupg git
This will install the development package group, as well as some extra utilities that the kernel build needs but are not part of this group.

The primary way through which I see in my dotfiles when there's a new kernel and then fetch its source code, is by using the JSON file that the kernel team presents on their website. You can get the latest stable kernel version using the following command:
curl "https://www.kernel.org/releases.json" | jq -r '.releases[1].version'
The source code archive for this kernel can be retrieved with the following command:
wget $(curl "https://www.kernel.org/releases.json" | jq -r '.releases[1].source')
Finally, the signature can be fetched using the following command:
wget $(curl "https://www.kernel.org/releases.json" | jq -r '.releases[1].pgp')

For now, grab the kernel source code and its signature. Additionally, as we're using the kernel version as reported by kernel.org in this tutorial a lot, assign a variable to it so that we have to disturb their server only once. Because this is a variable that only applies to the current shell, you'll have to set this again if you choose to log out during the build (e.g. if you're using SSH).
kver=$(curl "https://www.kernel.org/releases.json" | jq -r '.releases[1].version')

You can extract the kernel source code into a tarball using the following command:
unxz linux-${kver}.tar.xz

You should now verify the integrity of the kernel you've downloaded. Stable releases are always signed by Greg Kroah-Hartman's key. Its fingerprint is 647F 2865 4894 E3BD 4571 99BE 38DB BDC8 6092 693E as per kernel.org at the time of writing.
gpg --verify linux-${kver}.tar.sign

If the kernel signature matches, consider adding it to your trusted signatures by adding trust level 4 to it, and perhaps locally sign it. For this I'd like to point you at your favorite search engine as such actions can be irrevocable. Be sure to learn about PGP and GnuPG before doing this.

Next up, copy your tarball into tmpfs if you have sufficient memory available, this way the build will be done entirely from within RAM (with the ability to swap if needed and present). Ideally you'd have at least 4GB of RAM, of which 2GB is free to use. On systems with less than that, you might have to go into a TTY and work from there, and mount as much memory on /tmp as possible. Or just work from the hard drive instead (though you might still be able to compile from tmpfs if you add enough swap - every little bit of RAM helps).
To remount /tmp with more memory (replace 2G with the amount you need):
mount -o remount,size=2G /tmp
To copy the kernel into /tmp:
cp -v linux-${kver}.tar /tmp
Finally, change directory into /tmp.
cd /tmp

Now that we're in /tmp, extract the archive, and remove the original archive to save some space.
tar xf linux-${kver}.tar
rm linux-${kver}.tar
cd linux-${kver}

At this point, it would be beneficial to attach any hardware that you wish this new kernel to support. Alternatively, search on the Gentoo Handbook for the relevant article for whichever purposes that you wish your new kernel to support, that aren't in use right now. Examples would include OpenVPN, which needs a TUN driver.

Prepare the sources, and create an initial config.
make clean && make mrproper
make localmodconfig

Finally, add whichever extra options you need according to the Gentoo Handbook. Inside nconfig, you can also search for keywords inside the kernel configuration by hitting F8.
make nconfig
When you are finished adjusting the configuration, exit by hitting F9, and saving if needed.

At this point, we can actually start building our kernel.
make -j$(nproc)
Alternatively, you can put this in a screen session to put it in the background if you wish. Install screen first of course.
sudo pacman -S screen
screen -dmS kbuild make -j$(nproc)
See if the screen session still exists by using the following:
screen -ls
Or attach to it using the following:
screen -radU kbuild

When the kernel has finished building (this can take a while), continue by installing it.
Copy the kernel image:
sudo cp -v arch/$(uname -m)/boot/bzImage /boot/vmlinuz-${kver}
Install the modules:
sudo make modules_install
Install the headers:
sudo make headers_install
Generate the initramfs:
sudo mkinitcpio -k ${kver} -g /boot/initramfs-${kver}.img
Finally, rebuild the GRUB config.
sudo grub-mkconfig -o /boot/grub/grub.cfg

At this point it's time to reboot. Within GRUB, choose the advanced boot options and select your kernel from there. Otherwise, Arch will boot into the distribution kernel as it puts that first.

If you can successfully boot into your new kernel, you are done! You've now got yourself a system running your own homebrew kernel. Congratulations!

If you'd like to remove the distribution kernel to always boot into your own kernels by default, execute the following commands post-reboot.
sudo pacman -R linux
sudo grub-mkconfig -o /boot/grub/grub.cfg

If you want to upgrade into a new custom kernel from your current one, the same instructions apply again, but the difference is that at that time you can just reuse the config in /proc/config.gz. Thus, the localmodconfig can be replaced by this command instead: zcat /proc/config.gz > .config inside your kernel build directory. It is highly recommended to make periodic backups of this config file to ensure that after distro hops and such, you can hop right back in again without the need to reconfigure.

Thanks for reading this post. If you've got any more questions, feel free to ping me on Facebook or Telegram. I'm most active on Telegram. Please do not ping me if there's an appropriate group for it though. See catb's smart questions article to learn why.
On Telegram, you can also check out Silicon Network where I'm active in most groups. You can ask questions in the appropriate groups there. Be sure to do your own research first though.