mjhjmtu

I'm trying to run a recent enough kernel (4.9) on an Inforce 6540 board, a single-board computer with a Qualcomm Snapdragon 805 (Also called APQ8084, a SoC with four Krait 450 Cores, ARMv7-A).

The system starts, and print a trace on the serial interface, but can't run init.

Starting init: /sbin/init exists but couldn't execute it (error -117)

The error code may also be -2, in both case it makes no sense to me.

I also note that devtmpfs failed to mount. The ext4 rootfs is told to be mounted successfully.

Usual explanations are:

• 
  The kernel lacks devtmpfs support. I checked my config, devtmpfs is enabled.


• 
  The filesystem is corrupted. I replaced the kernel (and the devicetree) with the old custom 3.10 kernel originally used by the board, the old kernel manage to launch init (a link to systemd, which stops a little after, as we may expect from a userland built for a 4.9 kernel running on a 3.10 one). (the kernel and the devicetree are not on the rootfs partition, and are loaded by u-boot).

What may explain that init can't be run ?

Hereafter follows the whole boot trace from the kernel.

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.9.56+linaro (Jacen@dev-computer) (gcc version 6.2.1 20161016 (Linaro GCC 6.2-2016.11) ) #1 SMP PREEMPT Wed Apr 18 12:05:02 CEST 2018
[ 0.000000] CPU: ARMv7 Processor [513f06f1] revision 1 (ARMv7), cr=10c5787d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[ 0.000000] OF: fdt:Machine model: Qualcomm APQ8084/IFC6540
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] percpu: Embedded 15 pages/cpu @ee4b0000 s30796 r8192 d22452 u61440
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 784192
[ 0.000000] Kernel command line: root=/dev/mmcblk0p25 rw rootwait console=ttyMSM0,115200n8 androidboot.bootdevice=msm_sdcc.1 androidboot.serialno=a06b8142 androidboot.baseband=apq mdss_mdp.panel=0:dsi:0:
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 3100492K/3143680K available (8192K kernel code, 734K rwdata, 3176K rodata, 1024K init, 359K bss, 43188K reserved, 0K cma-reserved, 2359296K highmem)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
[ 0.000000] pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
[ 0.000000] modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
[ 0.000000] .text : 0xc0208000 - 0xc0b00000 (9184 kB)
[ 0.000000] .init : 0xc0f00000 - 0xc1000000 (1024 kB)
[ 0.000000] .data : 0xc1000000 - 0xc10b78a8 ( 735 kB)
[ 0.000000] .bss : 0xc10b9000 - 0xc1112cfc ( 360 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] arm_arch_timer: Architected cp15 and mmio timer(s) running at 19.20MHz (virt/virt).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[ 0.000005] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns
[ 0.000018] Switching to timer-based delay loop, resolution 52ns
[ 0.000200] Console: colour dummy device 80x30
[ 0.000223] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=192000)
[ 0.000238] pid_max: default: 32768 minimum: 301
[ 0.000363] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.000376] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.001050] CPU: Testing write buffer coherency: ok
[ 0.001312] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.001426] Setting up static identity map for 0x300000 - 0x300058
[ 0.200688] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.280884] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[ 0.361088] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[ 0.361323] Brought up 4 CPUs
[ 0.361347] SMP: Total of 4 processors activated (153.60 BogoMIPS).
[ 0.361357] CPU: All CPU(s) started in SVC mode.
[ 0.363838] devtmpfs: initialized
[ 0.368946] VFP support v0.3: implementor 51 architecture 64 part 6f variant 3 rev 1
[ 0.369232] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.369254] futex hash table entries: 1024 (order: 4, 65536 bytes)
[ 0.369431] pinctrl core: initialized pinctrl subsystem
[ 0.370514] NET: Registered protocol family 16
[ 0.370824] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.400854] cpuidle: using governor menu
[ 0.401142] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[ 0.401155] hw-breakpoint: maximum watchpoint size is 8 bytes.
[ 0.462767] vgaarb: loaded
[ 0.463148] SCSI subsystem initialized
[ 0.463639] usbcore: registered new interface driver usbfs
[ 0.463718] usbcore: registered new interface driver hub
[ 0.463830] usbcore: registered new device driver usb
[ 0.464664] Advanced Linux Sound Architecture Driver Initialized.
[ 0.466310] clocksource: Switched to clocksource arch_sys_counter
[ 0.533598] NET: Registered protocol family 2
[ 0.534156] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 0.534208] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.534285] TCP: Hash tables configured (established 8192 bind 8192)
[ 0.534341] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.534365] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.534527] NET: Registered protocol family 1
[ 0.535079] RPC: Registered named UNIX socket transport module.
[ 0.535092] RPC: Registered udp transport module.
[ 0.535103] RPC: Registered tcp transport module.
[ 0.535112] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.535438] Trying to unpack rootfs image as initramfs...
[ 0.535787] rootfs image is not initramfs (junk in compressed archive); looks like an initrd
[ 0.535885] Freeing initrd memory: 4K
[ 0.536245] hw perfevents: enabled with armv7_krait PMU driver, 5 counters available
[ 0.538614] workingset: timestamp_bits=14 max_order=20 bucket_order=6
[ 0.548667] NFS: Registering the id_resolver key type
[ 0.548700] Key type id_resolver registered
[ 0.548712] Key type id_legacy registered
[ 0.549461] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[ 0.550229] fuse init (API version 7.26)
[ 0.553343] bounce: pool size: 64 pages
[ 0.553401] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[ 0.553575] io scheduler noop registered
[ 0.553588] io scheduler deadline registered
[ 0.553779] io scheduler cfq registered (default)
[ 0.557749] msm_serial f995e000.serial: msm_serial: detected port #0
[ 0.557803] msm_serial f995e000.serial: uartclk = 7372800
[ 0.557852] f995e000.serial: ttyMSM0 at MMIO 0xf995e000 (irq = 170, base_baud = 460800) is a MSM
[ 0.557898] msm_serial: console setup on port #0
[ 1.127057] console [ttyMSM0] enabled
[ 1.132017] msm_serial: driver initialized
[ 1.135541] [drm] Initialized
[ 1.155600] brd: module loaded
[ 1.164058] loop: module loaded
[ 1.164452] SCSI Media Changer driver v0.25 
[ 1.166910] spmi spmi-0: PMIC Arb Version-1 (0x20000002)
[ 1.180872] libphy: Fixed MDIO Bus: probed
[ 1.181013] SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256) (6 bit encapsulation enabled).
[ 1.183860] CSLIP: code copyright 1989 Regents of the University of California.
[ 1.193649] usbcore: registered new interface driver ax88179_178a
[ 1.200800] usbcore: registered new interface driver cdc_ether
[ 1.207063] usbcore: registered new interface driver net1080
[ 1.212726] usbcore: registered new interface driver cdc_subset
[ 1.218601] usbcore: registered new interface driver cdc_ncm
[ 1.224574] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 1.230083] ehci-pci: EHCI PCI platform driver
[ 1.236461] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[ 1.241038] usbcore: registered new interface driver usbserial
[ 1.246351] usbcore: registered new interface driver usbserial_generic
[ 1.252217] usbserial: USB Serial support registered for generic
[ 1.260079] mousedev: PS/2 mouse device common for all mice
[ 1.265667] input: pm8941_pwrkey as /devices/platform/soc/fc4cf000.spmi/spmi-0/0-00/fc4cf000.spmi:pma8084@0:pwrkey@800/input/input0
[ 1.271688] rtc-pm8xxx fc4cf000.spmi:pma8084@0:rtc@6000: rtc core: registered pm8xxx_rtc as rtc0
[ 1.282117] i2c /dev entries driver
[ 1.291579] cpuidle: enable-method property 'qcom,kpss-acc-v2' found operations
[ 1.294356] cpuidle: enable-method property 'qcom,kpss-acc-v2' found operations
[ 1.301643] cpuidle: enable-method property 'qcom,kpss-acc-v2' found operations
[ 1.308978] cpuidle: enable-method property 'qcom,kpss-acc-v2' found operations
[ 1.316569] sdhci: Secure Digital Host Controller Interface driver
[ 1.323352] sdhci: Copyright(c) Pierre Ossman
[ 1.329630] sdhci-pltfm: SDHCI platform and OF driver helper
[ 1.406676] mmc0: SDHCI controller on f9824900.sdhci [f9824900.sdhci] using ADMA 64-bit
[ 1.417000] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.419371] usbcore: registered new interface driver usbhid
[ 1.424119] usbhid: USB HID core driver
[ 1.431805] usbcore: registered new interface driver snd-usb-audio
[ 1.444474] oprofile: using timer interrupt.
[ 1.445341] NET: Registered protocol family 10
[ 1.449328] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 1.452812] NET: Registered protocol family 17
[ 1.458042] Key type dns_resolver registered
[ 1.462550] Registering SWP/SWPB emulation handler
[ 1.490444] mmc0: new HS200 MMC card at address 0001
[ 1.491390] mmcblk0: mmc0:0001 SEM16G 14.7 GiB 
[ 1.495108] mmcblk0boot0: mmc0:0001 SEM16G partition 1 4.00 MiB
[ 1.504668] mmcblk0boot1: mmc0:0001 SEM16G partition 2 4.00 MiB
[ 1.505261] mmcblk0rpmb: mmc0:0001 SEM16G partition 3 4.00 MiB
[ 1.511106] rtc-pm8xxx fc4cf000.spmi:pma8084@0:rtc@6000: setting system clock to 1970-01-01 00:00:03 UTC (3)
[ 1.517337] ALSA device list:
[ 1.526473] No soundcards found.
[ 1.527644] Alternate GPT is invalid, using primary �
[ 1.539212] RAMDISK: Couldn't find valid RAM disk image starting at 0.
[ 1.540325] EXT4-fs (mmcblk0p25): couldn't mount as ext3 due to feature incompatibilities
[ 1.554971] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x400900
[ 1.584967] EXT4-fs (mmcblk0p25): recovery complete
[ 1.585980] EXT4-fs (mmcblk0p25): mounted filesystem with ordered data mode. Opts: (null)
[ 1.588779] VFS: Mounted root (ext4 filesystem) on device 179:25.
[ 1.597953] devtmpfs: error mounting -2
[ 1.605619] Freeing unused kernel memory: 1024K
[ 1.611168] random: fast init done
[ 1.613665] EXT4-fs error (device mmcblk0p25): ext4_lookup:1611: inode #2852: comm swapper/0: deleted inode referenced: 2952
[ 1.615789] Starting init: /sbin/init exists but couldn't execute it (error -117)
[ 1.631371] EXT4-fs error (device mmcblk0p25): ext4_lookup:1611: inode #2852: comm swapper/0: deleted inode referenced: 2952
[ 1.634427] Starting init: /bin/sh exists but couldn't execute it (error -117)
[ 1.644861] Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
[ 1.651777] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.9.56+linaro #1
[ 1.664870] Hardware name: Generic DT based system
[ 1.671400] [<c0310518>] (unwind_backtrace) from [<c030c684>] (show_stack+0x10/0x14)
[ 1.676166] [<c030c684>] (show_stack) from [<c061b97c>] (dump_stack+0x78/0x8c)
[ 1.684067] [<c061b97c>] (dump_stack) from [<c03edb40>] (panic+0xdc/0x250)
[ 1.691098] [<c03edb40>] (panic) from [<c0a4611c>] (kernel_init+0x104/0x114)
[ 1.697958] [<c0a4611c>] (kernel_init) from [<c03083b8>] (ret_from_fork+0x14/0x3c)
[ 1.705167] CPU1: stopping
[ 1.712535] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.9.56+linaro #1
[ 1.715220] Hardware name: Generic DT based system
[ 1.721754] [<c0310518>] (unwind_backtrace) from [<c030c684>] (show_stack+0x10/0x14)
[ 1.726525] [<c030c684>] (show_stack) from [<c061b97c>] (dump_stack+0x78/0x8c)
[ 1.734422] [<c061b97c>] (dump_stack) from [<c030f5b8>] (handle_IPI+0x34c/0x360)
[ 1.741452] [<c030f5b8>] (handle_IPI) from [<c03014a8>] (gic_handle_irq+0x88/0x8c)
[ 1.749011] [<c03014a8>] (gic_handle_irq) from [<c0a4bc4c>] (__irq_svc+0x6c/0xa8)
[ 1.756371] Exception stack(0xedce3f58 to 0xedce3fa0)
[ 1.763929] 3f40: 00000001 c0d2bda8
[ 1.768991] 3f60: 00000001 00000000 c1088754 c10a7a98 00000000 c1003098 65a28bb2 00000000
[ 1.777149] 3f80: 00000000 ed535c00 0682aaab edce3fa8 c0363868 c0840534 60000013 ffffffff
[ 1.785300] [<c0a4bc4c>] (__irq_svc) from [<c0840534>] (cpuidle_enter_state+0x154/0x428)
[ 1.793458] [<c0840534>] (cpuidle_enter_state) from [<c0363a90>] (cpu_startup_entry+0x148/0x218)
[ 1.801613] [<c0363a90>] (cpu_startup_entry) from [<0030154c>] (0x30154c)
[ 1.810366] CPU3: stopping
[ 1.817047] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.9.56+linaro #1
[ 1.819648] Hardware name: Generic DT based system
[ 1.826181] [<c0310518>] (unwind_backtrace) from [<c030c684>] (show_stack+0x10/0x14)
[ 1.830951] [<c030c684>] (show_stack) from [<c061b97c>] (dump_stack+0x78/0x8c)
[ 1.838852] [<c061b97c>] (dump_stack) from [<c030f5b8>] (handle_IPI+0x34c/0x360)
[ 1.845878] [<c030f5b8>] (handle_IPI) from [<c03014a8>] (gic_handle_irq+0x88/0x8c)
[ 1.853434] [<c03014a8>] (gic_handle_irq) from [<c0a4bc4c>] (__irq_svc+0x6c/0xa8)
[ 1.860798] Exception stack(0xedce7f58 to 0xedce7fa0)
[ 1.868352] 7f40: 00000003 c0d2bda8
[ 1.873417] 7f60: 00000001 00000000 c10887a4 c10a7a98 00000001 c1003098 65a41b46 00000000
[ 1.881577] 7f80: 00000001 ed574000 0682aaab edce7fa8 c0363868 c0840534 60000013 ffffffff
[ 1.889725] [<c0a4bc4c>] (__irq_svc) from [<c0840534>] (cpuidle_enter_state+0x154/0x428)
[ 1.897879] [<c0840534>] (cpuidle_enter_state) from [<c0363a90>] (cpu_startup_entry+0x148/0x218)
[ 1.906035] [<c0363a90>] (cpu_startup_entry) from [<0030154c>] (0x30154c)
[ 1.914792] CPU2: stopping
[ 1.921471] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.9.56+linaro #1
[ 1.924074] Hardware name: Generic DT based system
[ 1.930606] [<c0310518>] (unwind_backtrace) from [<c030c684>] (show_stack+0x10/0x14)
[ 1.935379] [<c030c684>] (show_stack) from [<c061b97c>] (dump_stack+0x78/0x8c)
[ 1.943277] [<c061b97c>] (dump_stack) from [<c030f5b8>] (handle_IPI+0x34c/0x360)
[ 1.950306] [<c030f5b8>] (handle_IPI) from [<c03014a8>] (gic_handle_irq+0x88/0x8c)
[ 1.957862] [<c03014a8>] (gic_handle_irq) from [<c0a4bc4c>] (__irq_svc+0x6c/0xa8)
[ 1.965224] Exception stack(0xedce5f58 to 0xedce5fa0)
[ 1.972780] 5f40: 00000002 c0d2bda8
[ 1.977843] 5f60: 00000001 00000000 c10887a4 c10a7a98 00000001 c1003098 65a5403e 00000000
[ 1.986003] 5f80: 00000001 ed535e00 0682aaab edce5fa8 c0363868 c0840534 60000013 ffffffff
[ 1.994149] [<c0a4bc4c>] (__irq_svc) from [<c0840534>] (cpuidle_enter_state+0x154/0x428)
[ 2.002307] [<c0840534>] (cpuidle_enter_state) from [<c0363a90>] (cpu_startup_entry+0x148/0x218)
[ 2.010464] [<c0363a90>] (cpu_startup_entry) from [<0030154c>] (0x30154c)
[ 2.019227] ---[ end Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.

I'd like to thanks the people who may have spent time on my issue, it turns out that the information in the question could not lead to the right root cause:

on some other boots, devtmpfs mounts properly, and init is killed instead of not being being executed, and there are some traces such as

[ 1.606597] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x400900

[ 1.636655] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

I expect the ext4 rootfs to be sane but the eMMC driver to be malfunctioning.

edit: the sdhci driver was unable to determine accurately the highest clock rate it could use. I fixed it to 200MHz and now my system boots correctly.