mjhjmtu

The hoster I am using provides a Debian 8.10 (kernel 4.9.85) as a rescue system. In the past I've been using that to bootstrap Ubuntu using debootstrap from here.

I am using a few preparational steps auch as installing apt-cacher-ng which is the reason for the localhost:3142 in the URL I am using (http://localhost:3142/us.archive.ubuntu.com/ubuntu/) and ubuntu-archive-keyring.

The debootstrap invocation is as follows:

LANG=C debootstrap --keep-debootstrap-dir --verbose --include=ubuntu-server,bash-completion,sudo,lshw,tmux,unzip,pciutils,usbutils,openssh-server,unattended-upgrades,linux-image-generic,cron --variant=minbase --arch=$(dpkg --print-architecture) bionic /target http://localhost:3142/us.archive.ubuntu.com/ubuntu/

(I added the --verbose only in the hope to see if anything is going wrong, to no avail.)

Now what's strange about the debootstrap run is that with this new version I end up seeing only the steps Retrieving, Validating and Extracting (for a subset of the packages) but nothing about these packages being set up.

So I thought to myself "alright, I did this with xenial, so let's try that again" and it gives me the same routine.

+ debootstrap --keep-debootstrap-dir --verbose --include=ubuntu-server,bash-completion,sudo,lshw,tmux,unzip,pciutils,usbutils,openssh-server,unattended-upgrades,linux-image-generic,cron --variant=minbase --arch=
amd64 xenial /target http://localhost:3142/us.archive.ubuntu.com/ubuntu/
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id 790BC7277767219C42C86F933B4FE6ACC0B21F32)
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Found additional base dependencies: acpid apport apt-utils at bcache-tools btrfs-tools busybox-initramfs byobu ca-certificates cloud-guest-utils cloud-initramfs-copymods cloud-initramfs-dyn-netconf cpio crda
cryptsetup cryptsetup-bin curl dh-python distro-info-data dmeventd dmsetup ethtool fonts-ubuntu-font-family-console gawk gcc-5-base gettext-base gir1.2-glib-2.0 git git-man gnupg gpgv grub-legacy-ec2 ifenslave i
fupdown initramfs-tools initramfs-tools-bin initramfs-tools-core iproute2 iso-codes iw klibc-utils kmod libapt-inst2.0 libapt-pkg5.0 libasn1-8-heimdal libasprintf0v5 libbsd0 libcurl3-gnutls libdbus-1-3 libdbus-g
lib-1-2 libdevmapper-event1.02.1 libdrm2 libdumbnet1 libedit2 liberror-perl libevent-2.0-5 libexpat1 libffi6 libfuse2 libgdbm3 libgirepository-1.0-1 libglib2.0-0 libgmp10 libgnutls30 libgpm2 libgssapi-krb5-2 lib
gssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhogweed4 libhx509-5-heimdal libicu55 libidn11 libk5crypto3 libkeyutils1 libklibc libkrb5-26-heimdal libkrb5-3 libkrb5support0 libl
dap-2.4-2 liblvm2app2.2 liblvm2cmd2.02 liblz4-1 liblzo2-2 libmnl0 libmpdec2 libmpfr4 libmspack0 libnettle6 libnewt0.52 libnl-3-200 libnl-genl-3-200 libp11-kit0 libpci3 libperl5.22 libplymouth4 libpng12-0 libpopt
0 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libpython3.5-minimal libpython3.5-stdlib libreadline5 libreadline6 libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules-db libsigs
egv2 libslang2 libsqlite3-0 libssl1.0.0 libstdc++6 libtasn1-6 libusb-0.1-4 libusb-1.0-0 libutempter0 libwind0-heimdal libwrap0 linux-base linux-firmware linux-image-4.4.0-21-generic linux-image-extra-4.4.0-21-ge
neric lsb-release lvm2 mdadm mime-support open-iscsi open-vm-tools openssh-client openssh-sftp-server openssl overlayroot patch perl perl-modules-5.22 plymouth python python-apt-common python-minimal python2.7 p
ython2.7-minimal python3 python3-apport python3-apt python3-chardet python3-dbus python3-debian python3-gi python3-minimal python3-newt python3-pkg-resources python3-problem-report python3-pycurl python3-six pyt
hon3-software-properties python3.5 python3.5-minimal readline-common screen software-properties-common sosreport ubuntu-cloudimage-keyring ubuntu-keyring ucf udev update-notifier-common vim vim-common vim-runtim
e vlan wireless-regdb xfsprogs xz-utils
I: Checking component main on http://localhost:3142/us.archive.ubuntu.com/ubuntu...
I: Retrieving acpid 1:2.0.26-1ubuntu2
I: Validating acpid 1:2.0.26-1ubuntu2
[...]
I: Chosen extractor for .deb packages: dpkg-deb
I: Extracting adduser...
I: Extracting base-files...
I: Extracting base-passwd...
[...]
I: Extracting zlib1g...

The reason this is odd is because in the past the setup stage worked without a hitch. But now it just gets silently skipped?! An older debootstrap version won't do, because those don't know about Bionic Beaver (Ubuntu 18.04).

There were two things I thought could be an issue:

1. the kernel version discrepancy: 4.15 in Ubuntu 18.04 versus 4.9.85 in Debian 8.10.


2. the libc version discrepancy: 2.27-3ubuntu1 versus 2.19-18+deb8u10.

... but in either case I'd expect some kind of error message from debootstrap. Also for xenial (Ubuntu 16.04) I shouldn't have to expect the same discrepancies. But I don't see any errors, instead the vital setup step gets skipped and attempting to chroot into /target with command /bin/bash just gives

# LANG=en_US.UTF-8 chroot /target /bin/bash
groups: cannot find name for group ID 0
I have no name!@rescue:/#

Digging a bit turns up no /etc/passwd and so on. /dev, /proc and /sys are bind-mounted into /target.

How can I troubleshoot this issue in order to successfully bootstrap a Ubuntu from said Debian rescue system? The architecture matches host and target, so what's preventing the setup step from being run?

NB: I cannot boot into a more recent kernel. Or rather the only way of achieving something like that would be by installing some kind of local rescue system first.

The software I am using

$ debootstrap --version
debootstrap 1.0.95ubuntu1
$ uname -a
Linux rescue 4.9.85 #2 SMP Thu Mar 1 08:06:18 CET 2018 x86_64 GNU/Linux

I also installed ubuntu-archive-keyring.

What else I tried

I also tried passing --foreign to debootstrap, which should cause the behavior I am seeing, but should also leave an executable /debootstrap/debootstrap which I can then invoke with --second-stage. However, while it exhibits the exact same behavior I am seeing without the --foreign command line switch, there is no /debootstrap/debootstrap in the target file system to complete the bootstrapping.

Furthermore I tried to use debootstrap from jessie-backports by installing it this way: apt-get install -t jessie-backports debootstrap (it identifies as debootstrap 1.0.89~bpo8+1). And then linking /usr/share/debootstrap/scripts/bionic to /usr/share/debootstrap/scripts/gutsy.

Alright I figured it out. The invocation of debootstrap did indeed indicate failure by returning an exit code of 1. I missed that because of how I was chaining several commands and using subshells.

Once I had figured that out, I had to find what issue debootstrap was encountering. It wasn't obvious from the /debootstrap/debootstrap.log (inside the target), although in hindsight it is. So I needed introspection on debootstrap. In order to do that invoked the /usr/sbin/debootstrap shell script explicitly via /bin/sh -x (to turn on tracing) and the output can actually be seen inside /debootstrap/debootstrap.log (inside the target). In my case the problem was mknod as shown by this trace entry and the output:

+ mknod -m 666 /target/dev/null c 1 3
mknod: '/target/dev/null': File exists

which in turn was caused by me mounting /dev, /proc and /sys into the target up front (something that had worked in the past!).

In new debootstrap versions this seems to fail unconditionally. The respective function calling mknod is setup_devices_simple from /usr/share/debootstrap/functions.

One notable change I saw from comparing the debootstrap scripts for 1.0.59 and 1.0.89~bpo8+1 (and subsequent versions) was that the invocation of the function setup_devices moved from function second_stage_install into function first_stage_install (of the gutsy script). This function calls setup_devices_simple unless doing variant fakechroot or running on a "foreign" kernel and therefore causes mknod to be invoked quite a bit earlier than before.

The reason why this didn't fail before, it seems, is because in the past the devices where kept inside a .tgz file and unpacked into place, whereas now debootstrap uses mkdnod directly.