I don't have enough confidence to do this alone and risk the server not to boot or something.

I would like to upgrade kernel from:

$ uname -r
4.9.0-6-amd64

$ uname -v
#1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07)

to kernel version 4.15 or 4.16. Whichever you recommend.

I just think I know how to list versions available:

$ apt-cache search linux-image | grep amd64

linux-headers-4.9.0-6-amd64 - Header files for Linux 4.9.0-6-amd64
linux-headers-4.9.0-6-rt-amd64 - Header files for Linux 4.9.0-6-rt-amd64
linux-image-4.9.0-6-amd64 - Linux 4.9 for 64-bit PCs
linux-image-4.9.0-6-amd64-dbg - Debug symbols for linux-image-4.9.0-6-amd64
linux-image-4.9.0-6-rt-amd64 - Linux 4.9 for 64-bit PCs, PREEMPT_RT
linux-image-4.9.0-6-rt-amd64-dbg - Debug symbols for linux-image-4.9.0-6-rt-amd64
linux-image-amd64 - Linux for 64-bit PCs (meta-package)
linux-image-amd64-dbg - Debugging symbols for Linux amd64 configuration (meta-package)
linux-image-rt-amd64 - Linux for 64-bit PCs (meta-package), PREEMPT_RT
linux-image-rt-amd64-dbg - Debugging symbols for Linux rt-amd64 configuration (meta-package)
linux-headers-4.9.0-3-amd64 - Header files for Linux 4.9.0-3-amd64
linux-headers-4.9.0-3-rt-amd64 - Header files for Linux 4.9.0-3-rt-amd64
linux-headers-4.9.0-4-amd64 - Header files for Linux 4.9.0-4-amd64
linux-headers-4.9.0-4-rt-amd64 - Header files for Linux 4.9.0-4-rt-amd64
linux-headers-4.9.0-5-amd64 - Header files for Linux 4.9.0-5-amd64
linux-headers-4.9.0-5-rt-amd64 - Header files for Linux 4.9.0-5-rt-amd64
linux-image-4.9.0-3-amd64 - Linux 4.9 for 64-bit PCs
linux-image-4.9.0-3-amd64-dbg - Debug symbols for linux-image-4.9.0-3-amd64
linux-image-4.9.0-3-rt-amd64 - Linux 4.9 for 64-bit PCs, PREEMPT_RT
linux-image-4.9.0-3-rt-amd64-dbg - Debug symbols for linux-image-4.9.0-3-rt-amd64
linux-image-4.9.0-4-amd64 - Linux 4.9 for 64-bit PCs
linux-image-4.9.0-4-amd64-dbg - Debug symbols for linux-image-4.9.0-4-amd64
linux-image-4.9.0-4-rt-amd64 - Linux 4.9 for 64-bit PCs, PREEMPT_RT
linux-image-4.9.0-4-rt-amd64-dbg - Debug symbols for linux-image-4.9.0-4-rt-amd64
linux-image-4.9.0-5-amd64 - Linux 4.9 for 64-bit PCs
linux-image-4.9.0-5-amd64-dbg - Debug symbols for linux-image-4.9.0-5-amd64
linux-image-4.9.0-5-rt-amd64 - Linux 4.9 for 64-bit PCs, PREEMPT_RT
linux-image-4.9.0-5-rt-amd64-dbg - Debug symbols for linux-image-4.9.0-5-rt-amd64
linux-headers-4.15.0-0.bpo.2-amd64 - Header files for Linux 4.15.0-0.bpo.2-amd64
linux-headers-4.15.0-0.bpo.2-cloud-amd64 - Header files for Linux 4.15.0-0.bpo.2-cloud-amd64
linux-headers-4.16.0-0.bpo.1-amd64 - Header files for Linux 4.16.0-0.bpo.1-amd64
linux-headers-4.16.0-0.bpo.1-cloud-amd64 - Header files for Linux 4.16.0-0.bpo.1-cloud-amd64
linux-image-4.15.0-0.bpo.2-amd64 - Linux 4.15 for 64-bit PCs
linux-image-4.15.0-0.bpo.2-amd64-dbg - Debug symbols for linux-image-4.15.0-0.bpo.2-amd64
linux-image-4.15.0-0.bpo.2-cloud-amd64 - Linux 4.15 for x86-64 cloud
linux-image-4.15.0-0.bpo.2-cloud-amd64-dbg - Debug symbols for linux-image-4.15.0-0.bpo.2-cloud-amd64
linux-image-4.16.0-0.bpo.1-amd64 - Linux 4.16 for 64-bit PCs
linux-image-4.16.0-0.bpo.1-amd64-dbg - Debug symbols for linux-image-4.16.0-0.bpo.1-amd64
linux-image-4.16.0-0.bpo.1-cloud-amd64 - Linux 4.16 for x86-64 cloud
linux-image-4.16.0-0.bpo.1-cloud-amd64-dbg - Debug symbols for linux-image-4.16.0-0.bpo.1-cloud-amd64
linux-headers-4.9.0-4-grsec-amd64 - Header files for Linux 4.9.0-4-grsec-amd64
linux-image-4.9.0-4-grsec-amd64 - Linux 4.9 for 64-bit PCs, Grsecurity protection (unofficial patch)
linux-image-grsec-amd64 - Linux image meta-package, grsec featureset
linux-image-cloud-amd64 - Linux for x86-64 cloud (meta-package)
linux-image-cloud-amd64-dbg - Debugging symbols for Linux cloud-amd64 configuration (meta-package)

I need headers too. On Ubuntu there is also package called extra or similarly, so I am confused not to see it here.

What is the proper way of installing new kernel manually on Debian 9?

If you want to install a newer Debian-packaged kernel, you should use one from the backports repository. You seem to have that repository already added to your apt configuration, so you're all set.

Since your current kernel is the basic amd64 version, I assume you won't need the realtime scheduler version, nor the cloud version.

Just run

apt-get install linux-image-4.16.0-0.bpo.1-amd64 linux-headers-4.16.0-0.bpo.1-amd64

i.e. "install the basic -amd64 version of the 4.16 kernel backported for Debian 9, and the corresponding headers package".

Unlike for regular packages, the new version linux-image package will not outright replace the existing 4.9.0 kernel, but will install alongside it. (That's because the version number is included as part of the package name.) The bootloaders will automatically be configured at linux-image post-install to either present the available kernels in a version-number-based order, or if that is not possible for some bootloaders, just automatically set the most recently installed one as the preferred one.

If it turns out that your new kernel won't boot, you can just select the previous kernel from the bootloader, and then remove the kernel package that proved to be non-functional. If you accidentally tell the package manager to remove the kernel you're currently running on, it is smart enough to know that isn't a good thing to do, and will abort the operation.

Issue at Hand

You require a newer Linux kernel than the one installed on your Debian instance. There are some risks to this. It is recommended that you avoid straying from the default stable LTS kernel that Debian provides. However due to needs on your end you require a newer kernel.

Solution

You can add the backports repository, use apt-pinning to control what is installed here, and install the necessary components to upgrade your kernel.

I highly recommend apt-pinning as you can add alternate repositories or even third-party repositories and control what is installed from them. This allows you, again at your own risk, to install and update alternative packages and libraries and update them along side the main components of your Operating System. Apt-pinning allows for granular control of packages from alternate sources (i.e. not stable) with lower risks to breakage. If you understand the risk you can move forward with this.

Add Backports repository and set up apt-preferences

First step is to add the proper repositories to your sources.list. I would recommend you use /sources.list.d/ but you can do this in just one file.

#Main Stable Repo
deb http://deb.debian.org stable main contrib non-free 
deb-src http://deb.debian.org stable main contrib non-free
#Stable-updates
deb http://deb.debian.org stable-updates main contrib non-free 
deb-src http://deb.debian.org stable-updates main contrib non-free 
#Security Updates 
deb http://deb.debian.org/debian-secruity stable/updates main contrib non-free
deb-src http://deb.debian.org/debian-secruity stable/updates main contrib non-free 
#Stretch Backports 
deb http://ftp.debian.org/debian stretch-backports main contrib non-free
deb-src http://ftp.debian.org/debian stretch-backports main contrib non-free

Your sources.list should look something like this. You do not need the non-free parts if you do not require non-free software. Next you will need to give packages pin priorities using /etc/apt/preferences. This file should look like this:

Package: *
Pin: release a=debian-security
Pin-Priority: 1000

Package: *
Pin: release a=stable
Pin-Priority: 900

Package: *
Pin: release a=stable-updates
Pin-Priority: 800

Package: *
Pin: release a=stretch-backports
Pin-Priority: 700

You can be more granular in your approach specifying individual packages, and assigning higher or lower values to the pin-priorities.

After this you run apt-get update.

Install Desired Kernel

Using apt-cache search linux-image you will list all available kernels. When you have identified which kernel you want you can install it using apt-get install linux-image-flavour. It is recommended that you install the matching Linux header package as well. If you require the grsec image in your environment install that one instead. Again with the appropriate header package as well.

Upon reboot you will be able to select which kernel you want to launch at the GRUB menu. If you run into issues you can always select the known good kernel, and boot using that instead.

Conclusion

I have used apt-pinning to successful manage my personal computer's Debian install. I used the above method to install what was the latest kernel at the time (4.15) and my system has been working fine. However take this with a grain of salt as it is a personal computer and not what I assume is a production server you wish to install this on. I am including links to all the sources I referenced as well. I used this Debian Wiki page to inform on the necessary steps, as well as the page on apt-pinning. This article also was referenced.

If you have any questions or concerns about this post, do not hesitate to ask me. If there are any corrections or misconceptions in this answer please inform me. I can update the post as necessary.

Best of Luck!

To install the linux-image and its headers package use the following command:

apt install linux-image,headers-4.16.0-0.bpo.1-amd64

Some dependencies need to be installed from backports after running the above command , it can be installed through :

apt install -t stretch-backports pckg_name