Determining cause of kernel panics

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite
1












I have a HP DL360 G7 running CentOS Linux release 7.4.1708 (Core) and suffering from kernel panics at random intervals and (to my eyes) random reasons. I've dug into the crash dumps using these as a guides: https://www.slideshare.net/PaulVNovarese/linux-crash-dump-capture-and-analysis and https://www.dedoimedo.com/computers/crash-analyze.html



But the crash dumps seem to my eyes pointing to a different reason each time, i.e, I don't have the competency to understand crash dumps well enough to determine if they have any connection.



All the crash dumps that I have managed to get. Paste contains crash commands sys, bt and log of each dump.



EDIT

Due to suggested edits, I've pasted the sys and bt outputs of crash command. I'll keep the pastebin links as the log output of crash is too large to fit here.



Also adding that the server is running ECC memory and no indication of MCEs appear in logs so I'm not convinced that this is a faulty memory situation.



Any help or guide how to solve this would be greatly appreciated.



https://pastebin.ca/3955184



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2018-01-03-03:08:48/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Wed Jan 3 03:07:40 2018
 UPTIME: 28 days, 00:57:45
LOAD AVERAGE: 3.45, 2.43, 2.66
 TASKS: 714
 NODENAME: server
 RELEASE: 3.10.0-693.11.1.el7.x86_64
 VERSION: #1 SMP Mon Dec 4 23:52:40 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "general protection fault: 0000 [#1] SMP "


btoutput



PID: 24892 TASK: ffff8808f9111fa0 CPU: 0 COMMAND: "python"
 #0 [ffff8808fba03910] machine_kexec at ffffffff8105c52b
 #1 [ffff8808fba03970] __crash_kexec at ffffffff81104a42
 #2 [ffff8808fba03a40] crash_kexec at ffffffff81104b30
 #3 [ffff8808fba03a58] oops_end at ffffffff816ad338
 #4 [ffff8808fba03a80] die at ffffffff8102e97b
 #5 [ffff8808fba03ab0] do_general_protection at ffffffff816accbe
 #6 [ffff8808fba03ae0] general_protection at ffffffff816ac568
 [exception RIP: inet6_csk_search_req+261]
 RIP: ffffffff81673385 RSP: ffff8808fba03b98 RFLAGS: 00010202
 RAX: 0000000000001c9e RBX: 3932383931333431 RCX: ffff8807ed7918c8
 RDX: 00000000ffffffff RSI: 0000000000000000 RDI: 00000000b1fa562e
 RBP: ffff8808fba03bb0 R8: ffff8807ed7918d8 R9: 0000000000000001
 R10: ffff8801f3ffc000 R11: 00000000b80dcf9c R12: ffff880783d27178
 R13: ffff8808fba03bd0 R14: ffff8808ec21a9a8 R15: 0000000000000000
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #7 [ffff8808fba03bb8] tcp_v6_do_rcv at ffffffff8166d73a
 #8 [ffff8808fba03c10] tcp_v6_rcv at ffffffff8166e1b2
 #9 [ffff8808fba03cc0] ip6_input_finish at ffffffff81643712
#10 [ffff8808fba03d00] ip6_input at ffffffff81643fe3
#11 [ffff8808fba03d58] ip6_rcv_finish at ffffffff81643518
#12 [ffff8808fba03d70] ipv6_rcv at ffffffff81643d99
#13 [ffff8808fba03df0] __netif_receive_skb_core at ffffffff81586f22
#14 [ffff8808fba03e60] __netif_receive_skb at ffffffff81587188
#15 [ffff8808fba03e80] process_backlog at ffffffff8158841e
#16 [ffff8808fba03ec0] net_rx_action at ffffffff8158799d
#17 [ffff8808fba03f40] __do_softirq at ffffffff81090b4f
#18 [ffff8808fba03fb0] call_softirq at ffffffff816b6b1c
--- <IRQ stack> ---
#19 [ffff880c1fcb3980] local_bh_enable at ffffffff81090017
#20 [ffff880c1fcb3990] __dev_queue_xmit at ffffffff815895a5
#21 [ffff880c1fcb39e8] local_bh_enable at ffffffff81090017
#22 [ffff880c1fcb39f8] ip6_finish_output2 at ffffffff816407b1
#23 [ffff880c1fcb3a78] ip6_finish_output at ffffffff81642cbc
#24 [ffff880c1fcb3aa0] ip6_output at ffffffff81642d77
#25 [ffff880c1fcb3b00] ip6_xmit at ffffffff81640039
#26 [ffff880c1fcb3ba8] inet6_csk_xmit at ffffffff81673059
#27 [ffff880c1fcb3c48] tcp_transmit_skb at ffffffff815e7c9f
#28 [ffff880c1fcb3cb8] tcp_connect at ffffffff815e97ed
#29 [ffff880c1fcb3d38] tcp_v6_connect at ffffffff8166c106
#30 [ffff880c1fcb3e08] __inet_stream_connect at ffffffff81605725
#31 [ffff880c1fcb3e80] inet_stream_connect at ffffffff816059d8
#32 [ffff880c1fcb3eb0] SYSC_connect at ffffffff8156a497
#33 [ffff880c1fcb3f70] sys_connect at ffffffff8156b29e
#34 [ffff880c1fcb3f80] system_call_fastpath at ffffffff816b5089
 RIP: 00007f9e58f319d0 RSP: 00007ffece4029e0 RFLAGS: 00010246
 RAX: 000000000000002a RBX: ffffffff816b5089 RCX: 00007f9e59144c28
 RDX: 000000000000001c RSI: 00007ffece4030d0 RDI: 0000000000000003
 RBP: 0000000000000000 R8: 0000000000000000 R9: 00000000032dfb60
 R10: 0000000000000006 R11: 0000000000000246 R12: ffffffff8156b29e
 R13: ffff880c1fcb3f78 R14: 0000000002b56510 R15: 00000000025950a0
 ORIG_RAX: 000000000000002a CS: 0033 SS: 002b


https://pastebin.ca/3955185 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-12-06-02:05:41/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Wed Dec 6 02:04:35 2017
 UPTIME: 15 days, 00:23:09
LOAD AVERAGE: 6.23, 4.95, 3.77
 TASKS: 726
 NODENAME: server
 RELEASE: 3.10.0-693.5.2.el7.x86_64
 VERSION: #1 SMP Fri Oct 20 20:32:50 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 00000000000008d0"


bt output



PID: 31570 TASK: ffff8811f9f9dee0 CPU: 11 COMMAND: "mysqld"
 #0 [ffff8800988ff340] machine_kexec at ffffffff8105c4cb
 #1 [ffff8800988ff3a0] __crash_kexec at ffffffff81104a42
 #2 [ffff8800988ff470] crash_kexec at ffffffff81104b30
 #3 [ffff8800988ff488] oops_end at ffffffff816ad338
 #4 [ffff8800988ff4b0] no_context at ffffffff8169d35a
 #5 [ffff8800988ff500] __bad_area_nosemaphore at ffffffff8169d3f0
 #6 [ffff8800988ff548] bad_area at ffffffff8169d714
 #7 [ffff8800988ff570] __do_page_fault at ffffffff816b02fc
 #8 [ffff8800988ff5d0] do_page_fault at ffffffff816b03a5
 #9 [ffff8800988ff600] page_fault at ffffffff816ac5c8
 [exception RIP: xfs_fs_destroy_inode+78]
 RIP: ffffffffc03d6f3e RSP: ffff8800988ff6b0 RFLAGS: 00010202
 RAX: 0000000000000000 RBX: ffff8809781d00f4 RCX: 0000000000020100
 RDX: 000000000000000b RSI: ffff8809781d01d8 RDI: ffff8809781d0000
 RBP: ffff8800988ff6c8 R8: 0000000000000000 R9: 09781d01f00c0000
 R10: f669e55c17347c03 R11: 0000000000000000 R12: ffff8809781d0000
 R13: ffff8809781d0150 R14: ffff8811f97c2108 R15: ffff880cd7cbfe08
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff8800988ff6d0] evict at ffffffff8121d658
#11 [ffff8800988ff710] iget_locked at ffffffff8121d83e
#12 [ffff8800988ff738] prune_icache_sb at ffffffff8121e834
#13 [ffff8800988ff7f0] vmpressure at ffffffff811f7451
#14 [ffff8800988ff878] do_try_to_free_pages at ffffffff811985a2
#15 [ffff8800988ff8f0] try_to_free_pages at ffffffff811987bc
#16 [ffff8800988ff988] __alloc_pages_slowpath at ffffffff8169fbcb
#17 [ffff8800988ffa78] __alloc_pages_nodemask at ffffffff8118cdb5
#18 [ffff8800988ffb28] alloc_pages_current at ffffffff811d1078
#19 [ffff8800988ffb70] alloc_skb_with_frags at ffffffff81573cbd
#20 [ffff8800988ffbc0] sock_alloc_send_pskb at ffffffff8156c9b9
#21 [ffff8800988ffc48] unix_stream_sendmsg at ffffffff8163aef0
#22 [ffff8800988ffcf8] sock_sendmsg at ffffffff8156a580
#23 [ffff8800988ffe58] SYSC_sendto at ffffffff8156a731
#24 [ffff8800988fff70] sys_sendto at ffffffff8156b2ce
#25 [ffff8800988fff80] system_call_fastpath at ffffffff816b5089
 RIP: 00007fca94f07c0b RSP: 00007fca7fed7308 RFLAGS: 00000202
 RAX: 000000000000002c RBX: ffffffff816b5089 RCX: 00000000014c8a7c
 RDX: 0000000000004000 RSI: 00007fca6ebdb008 RDI: 0000000000000163
 RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000000
 R10: 0000000000000040 R11: 0000000000000246 R12: ffffffff8156b2ce
 R13: ffff8800988fff78 R14: 00007fca6ebef238 R15: 0000000000004000
 ORIG_RAX: 000000000000002c CS: 0033 SS: 002b


https://pastebin.ca/3955186 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-14-04:21:30/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Tue Nov 14 04:20:23 2017
 UPTIME: 09:25:26
LOAD AVERAGE: 2.21, 1.47, 1.71
 TASKS: 788
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008"


bt output



PID: 29737 TASK: ffff8811f6346eb0 CPU: 3 COMMAND: "kworker/3:0"
 #0 [ffff8811ac617948] machine_kexec at ffffffff8105c4cb
 #1 [ffff8811ac6179a8] __crash_kexec at ffffffff81104a32
 #2 [ffff8811ac617a78] crash_kexec at ffffffff81104b20
 #3 [ffff8811ac617ab8] no_context at ffffffff8169d2ba
 #4 [ffff8811ac617b08] no_context at ffffffff8169d350
 #5 [ffff8811ac617b50] __bad_area_nosemaphore at ffffffff8169d4ba
 #6 [ffff8811ac617b60] __do_page_fault at ffffffff816b017e
 #7 [ffff8811ac617bc0] __do_page_fault at ffffffff816b0325
 #8 [ffff8811ac617bf0] general_protection at ffffffff816ac548
 [exception RIP: xlog_write+772]
 RIP: ffffffffc03f8644 RSP: ffff8811ac617ca0 RFLAGS: 00010286
 RAX: 0000000000000000 RBX: ffff8811af615870 RCX: 0000000000000000
 RDX: 00000000000261a0 RSI: 0000000000000000 RDI: 0000000000005e6c
 RBP: ffff8811ac617d38 R8: 0000000000000000 R9: ffffc9000d576078
 R10: ffffc9000d57606c R11: ffff8808f53ad800 R12: 0000000000007e00
 R13: 00000000000000d0 R14: 0000000000000000 R15: ffff880c9c7bb000
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff8811ac617d40] xlog_cil_push at ffffffffc03fa1f8 [xfs]
#10 [ffff8811ac617e10] xlog_cil_push_work at ffffffffc03fa395 [xfs]
#11 [ffff8811ac617e20] process_one_work at ffffffff810a881a
#12 [ffff8811ac617e68] worker_thread at ffffffff810a94e6
#13 [ffff8811ac617ec8] kthread at ffffffff810b098f
#14 [ffff8811ac617f50] save_rest at ffffffff816b4f58


https://pastebin.ca/3955187 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-13-18:50:12/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Mon Nov 13 18:49:05 2017
 UPTIME: 10 days, 17:30:03
LOAD AVERAGE: 2.10, 2.43, 2.33
 TASKS: 785
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000020"


btoutput



PID: 27951 TASK: ffff8808f1b25ee0 CPU: 16 COMMAND: "php-cgi"
 #0 [ffff880113643a90] machine_kexec at ffffffff8105c4cb
 #1 [ffff880113643af0] __crash_kexec at ffffffff81104a32
 #2 [ffff880113643bc0] crash_kexec at ffffffff81104b20
 #3 [ffff880113643c00] no_context at ffffffff8169d2ba
 #4 [ffff880113643c50] no_context at ffffffff8169d350
 #5 [ffff880113643c98] mm_fault_error at ffffffff8169d674
 #6 [ffff880113643cc0] __do_page_fault at ffffffff816b027c
 #7 [ffff880113643d20] __do_page_fault at ffffffff816b0325
 #8 [ffff880113643d50] general_protection at ffffffff816ac548
 [exception RIP: xfs_free_eofblocks+91]
 RIP: ffffffffc03cf8db RSP: ffff880113643e00 RFLAGS: 00010212
 RAX: 0000000000000000 RBX: ffff8811f3bdcec0 RCX: 000000000000000c
 RDX: 0000000000000fff RSI: 0000000000000001 RDI: ffff8811f3bdcec0
 RBP: ffff880113643e58 R8: 0000000000000000 R9: 0000000000000000
 R10: ffff8811f3bdd010 R11: ffff880017c7b710 R12: 0000000000015a0b
 R13: ffff8808f7346000 R14: ffff8808f4ba00c0 R15: ffff880170ad2d20
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff880113643e60] xfs_release at ffffffffc03e7105 [xfs]
#10 [ffff880113643e88] xfs_file_release at ffffffffc03d65a5 [xfs]
#11 [ffff880113643e98] delayed_fput at ffffffff81202fc9
#12 [ffff880113643ee0] alloc_file at ffffffff8120322e
#13 [ffff880113643ef0] task_work_run at ffffffff810ad247
#14 [ffff880113643f30] do_notify_resume at ffffffff8102ab62
#15 [ffff880113643f50] int_with_check at ffffffff816b52bd
 RIP: 00007fd2f72abe90 RSP: 00007ffe5d25d848 RFLAGS: 00000246
 RAX: 0000000000000000 RBX: 000055bac5cda3c0 RCX: ffffffffffffffff
 RDX: 0000000000000008 RSI: 0000000000000001 RDI: 0000000000000003
 RBP: 0000000000000000 R8: 000055bac5cda4a0 R9: 00007fd2fa6bb840
 R10: 000000000000000d R11: 0000000000000246 R12: 0000000000000000
 R13: 0000000000000001 R14: 00007fd2e8771f58 R15: 0000000000000000
 ORIG_RAX: 0000000000000003 CS: 0033 SS: 002b


https://pastebin.ca/3955188 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-03-01:14:25/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Fri Nov 3 01:13:19 2017
 UPTIME: 3 days, 18:53:59
LOAD AVERAGE: 2.66, 2.58, 2.16
 TASKS: 744
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at (null)"


bt output



PID: 144 TASK: ffff8808faf52f70 CPU: 7 COMMAND: "kswapd1"
 #0 [ffff8808f9e0f7e8] machine_kexec at ffffffff8105c4cb
 #1 [ffff8808f9e0f848] __crash_kexec at ffffffff81104a32
 #2 [ffff8808f9e0f918] crash_kexec at ffffffff81104b20
 #3 [ffff8808f9e0f958] no_context at ffffffff8169d2ba
 #4 [ffff8808f9e0f9a8] no_context at ffffffff8169d350
 #5 [ffff8808f9e0f9f0] __bad_area_nosemaphore at ffffffff8169d4ba
 #6 [ffff8808f9e0fa00] __do_page_fault at ffffffff816b017e
 #7 [ffff8808f9e0fa08] __radix_tree_create at ffffffff81328a9e
 #8 [ffff8808f9e0fa60] __do_page_fault at ffffffff816b0325
 #9 [ffff8808f9e0fa90] general_protection at ffffffff816ac548
 [exception RIP: crc32_generic_combine+89]
 RIP: ffffffff8133db39 RSP: ffff8808f9e0fb40 RFLAGS: 00010207
 RAX: 0000000000000000 RBX: ffff88010b138008 RCX: dead000000000200
 RDX: 0000000000000000 RSI: ffff88010b8bc570 RDI: ffff88010b138008
 RBP: ffff8808f9e0fb40 R8: e018000000000000 R9: 010b8bc5700c0000
 R10: fed6747d72e55c03 R11: 0000000000000000 R12: ffff88010b138000
 R13: ffff8808f47dbcb8 R14: ffff8811f973f108 R15: ffff880143a76b48
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#10 [ffff8808f9e0fb48] file_has_perm at ffffffff812b752d
#11 [ffff8808f9e0fb88] evict at ffffffff8121d5e2
#12 [ffff8808f9e0fbb8] iget_locked at ffffffff8121d81a
#13 [ffff8808f9e0fbe0] iget_locked at ffffffff8121d8ce
#14 [ffff8808f9e0fc08] new_inode_pseudo at ffffffff8121e8c4
#15 [ffff8808f9e0fc70] ns_set_super at ffffffff81203878
#16 [ffff8808f9e0fca8] shrink_slab at ffffffff81195413
#17 [ffff8808f9e0fcc0] vmpressure_register_event at ffffffff811f7547
#18 [ffff8808f9e0fd48] balance_pgdat at ffffffff81199081
#19 [ffff8808f9e0fe20] kswapd at ffffffff81199323
#20 [ffff8808f9e0fe78] wake_up_atomic_t at ffffffff810b1910
#21 [ffff8808f9e0fea8] balance_pgdat at ffffffff811991b0
#22 [ffff8808f9e0fec8] kthread at ffffffff810b098f
#23 [ffff8808f9e0ff50] save_rest at ffffffff816b4f58

https://pastebin.ca/3955189

 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-10-30-07:14:38/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Mon Oct 30 06:13:31 2017
 UPTIME: 3 days, 06:21:18
LOAD AVERAGE: 1.03, 1.19, 1.36
 TASKS: 707
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "general protection fault: 0000 [#1] SMP "


bt output



PID: 32005 TASK: ffff8808f8bfeeb0 CPU: 4 COMMAND: "vtund"
 #0 [ffff8804623fb950] machine_kexec at ffffffff8105c4cb
 #1 [ffff8804623fb9b0] __crash_kexec at ffffffff81104a32
 #2 [ffff8804623fba80] crash_kexec at ffffffff81104b20
 #3 [ffff8804623fbac0] die at ffffffff8102e97b
 #4 [ffff8804623fbaf0] do_general_protection at ffffffff816acc3e
 #5 [ffff8804623fbb20] xen_int3 at ffffffff816ac4e8
 [exception RIP: memcmp]
 RIP: ffffffff8132b980 RSP: ffff8804623fbbd0 RFLAGS: 00010286
 RAX: ffff8801f3cd7048 RBX: 000000000000000a RCX: ffff8806917dc048
 RDX: ffff8808fb333800 RSI: ffff8808f9f482e8 RDI: 0236303633043a00
 RBP: ffff8804623fbc40 R8: 0000000000000036 R9: 0000000000000000
 R10: ffff88017fc03400 R11: 0000000000000000 R12: ffffffff8197ea30
 R13: 0236303633043a00 R14: ffff8808fb333868 R15: ffff8808a4f6a300
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
 #6 [ffff8804623fbbd0] insert_header at ffffffff8127cb99
 #7 [ffff8804623fbc48] register_leaf_sysctl_tables at ffffffff8127d28b
 #8 [ffff8804623fbc60] bstr_printf at ffffffff8132fb59
 #9 [ffff8804623fbd40] ipv6_add_dev at ffffffff8164784a
#10 [ffff8804623fbd70] addrconf_notify at ffffffff8164c9c9
#11 [ffff8804623fbd78] sysfs_slab_alias at ffffffff811df8d6
#12 [ffff8804623fbd88] dropmon_net_event at ffffffff815ab686
#13 [ffff8804623fbde0] trace_do_page_fault at ffffffff816b051c
#14 [ffff8804623fbe18] raw_notifier_call_chain at ffffffff810b68d6
#15 [ffff8804623fbe28] call_netdevice_notifiers_info at ffffffff8158306d
#16 [ffff8804623fbe50] register_netdevice at ffffffff8158c576
#17 [ffff8804623fbe88] __tun_chr_ioctl at ffffffffc07fca8e [tun]
#18 [ffff8804623fbf20] tun_chr_compat_ioctl at ffffffffc07fd10b [tun]
#19 [ffff8804623fbf30] compat_sys_ioctl at ffffffff8125d73b
#20 [ffff8804623fbf80] cstar_tracesys at ffffffff816b746c
 RIP: 00000000080d458c RSP: 00000000fff1980c RFLAGS: 00000246
 RAX: ffffffffffffffda RBX: ffffffff816b746c RCX: 00000000400454ca
 RDX: 00000000fff19840 RSI: 00000000fff19840 RDI: 00000000fff19860
 RBP: 00000000fff19878 R8: 0000000000000000 R9: 0000000000000000
 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 ORIG_RAX: 0000000000000036 CS: 0023 SS: 002b



centos kernel-panic




share|improve this question






















  • I suggest you directly put all the relevant information here. With proper formatting you can show all of them in a ordered manner without worrying that it would distract people from your main question.
    – Weijun Zhou
    Jan 4 at 3:25














up vote
1
down vote

favorite
1












I have a HP DL360 G7 running CentOS Linux release 7.4.1708 (Core) and suffering from kernel panics at random intervals and (to my eyes) random reasons. I've dug into the crash dumps using these as a guides: https://www.slideshare.net/PaulVNovarese/linux-crash-dump-capture-and-analysis and https://www.dedoimedo.com/computers/crash-analyze.html



But the crash dumps seem to my eyes pointing to a different reason each time, i.e, I don't have the competency to understand crash dumps well enough to determine if they have any connection.



All the crash dumps that I have managed to get. Paste contains crash commands sys, bt and log of each dump.



EDIT

Due to suggested edits, I've pasted the sys and bt outputs of crash command. I'll keep the pastebin links as the log output of crash is too large to fit here.



Also adding that the server is running ECC memory and no indication of MCEs appear in logs so I'm not convinced that this is a faulty memory situation.



Any help or guide how to solve this would be greatly appreciated.



https://pastebin.ca/3955184



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2018-01-03-03:08:48/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Wed Jan 3 03:07:40 2018
 UPTIME: 28 days, 00:57:45
LOAD AVERAGE: 3.45, 2.43, 2.66
 TASKS: 714
 NODENAME: server
 RELEASE: 3.10.0-693.11.1.el7.x86_64
 VERSION: #1 SMP Mon Dec 4 23:52:40 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "general protection fault: 0000 [#1] SMP "


btoutput



PID: 24892 TASK: ffff8808f9111fa0 CPU: 0 COMMAND: "python"
 #0 [ffff8808fba03910] machine_kexec at ffffffff8105c52b
 #1 [ffff8808fba03970] __crash_kexec at ffffffff81104a42
 #2 [ffff8808fba03a40] crash_kexec at ffffffff81104b30
 #3 [ffff8808fba03a58] oops_end at ffffffff816ad338
 #4 [ffff8808fba03a80] die at ffffffff8102e97b
 #5 [ffff8808fba03ab0] do_general_protection at ffffffff816accbe
 #6 [ffff8808fba03ae0] general_protection at ffffffff816ac568
 [exception RIP: inet6_csk_search_req+261]
 RIP: ffffffff81673385 RSP: ffff8808fba03b98 RFLAGS: 00010202
 RAX: 0000000000001c9e RBX: 3932383931333431 RCX: ffff8807ed7918c8
 RDX: 00000000ffffffff RSI: 0000000000000000 RDI: 00000000b1fa562e
 RBP: ffff8808fba03bb0 R8: ffff8807ed7918d8 R9: 0000000000000001
 R10: ffff8801f3ffc000 R11: 00000000b80dcf9c R12: ffff880783d27178
 R13: ffff8808fba03bd0 R14: ffff8808ec21a9a8 R15: 0000000000000000
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #7 [ffff8808fba03bb8] tcp_v6_do_rcv at ffffffff8166d73a
 #8 [ffff8808fba03c10] tcp_v6_rcv at ffffffff8166e1b2
 #9 [ffff8808fba03cc0] ip6_input_finish at ffffffff81643712
#10 [ffff8808fba03d00] ip6_input at ffffffff81643fe3
#11 [ffff8808fba03d58] ip6_rcv_finish at ffffffff81643518
#12 [ffff8808fba03d70] ipv6_rcv at ffffffff81643d99
#13 [ffff8808fba03df0] __netif_receive_skb_core at ffffffff81586f22
#14 [ffff8808fba03e60] __netif_receive_skb at ffffffff81587188
#15 [ffff8808fba03e80] process_backlog at ffffffff8158841e
#16 [ffff8808fba03ec0] net_rx_action at ffffffff8158799d
#17 [ffff8808fba03f40] __do_softirq at ffffffff81090b4f
#18 [ffff8808fba03fb0] call_softirq at ffffffff816b6b1c
--- <IRQ stack> ---
#19 [ffff880c1fcb3980] local_bh_enable at ffffffff81090017
#20 [ffff880c1fcb3990] __dev_queue_xmit at ffffffff815895a5
#21 [ffff880c1fcb39e8] local_bh_enable at ffffffff81090017
#22 [ffff880c1fcb39f8] ip6_finish_output2 at ffffffff816407b1
#23 [ffff880c1fcb3a78] ip6_finish_output at ffffffff81642cbc
#24 [ffff880c1fcb3aa0] ip6_output at ffffffff81642d77
#25 [ffff880c1fcb3b00] ip6_xmit at ffffffff81640039
#26 [ffff880c1fcb3ba8] inet6_csk_xmit at ffffffff81673059
#27 [ffff880c1fcb3c48] tcp_transmit_skb at ffffffff815e7c9f
#28 [ffff880c1fcb3cb8] tcp_connect at ffffffff815e97ed
#29 [ffff880c1fcb3d38] tcp_v6_connect at ffffffff8166c106
#30 [ffff880c1fcb3e08] __inet_stream_connect at ffffffff81605725
#31 [ffff880c1fcb3e80] inet_stream_connect at ffffffff816059d8
#32 [ffff880c1fcb3eb0] SYSC_connect at ffffffff8156a497
#33 [ffff880c1fcb3f70] sys_connect at ffffffff8156b29e
#34 [ffff880c1fcb3f80] system_call_fastpath at ffffffff816b5089
 RIP: 00007f9e58f319d0 RSP: 00007ffece4029e0 RFLAGS: 00010246
 RAX: 000000000000002a RBX: ffffffff816b5089 RCX: 00007f9e59144c28
 RDX: 000000000000001c RSI: 00007ffece4030d0 RDI: 0000000000000003
 RBP: 0000000000000000 R8: 0000000000000000 R9: 00000000032dfb60
 R10: 0000000000000006 R11: 0000000000000246 R12: ffffffff8156b29e
 R13: ffff880c1fcb3f78 R14: 0000000002b56510 R15: 00000000025950a0
 ORIG_RAX: 000000000000002a CS: 0033 SS: 002b


https://pastebin.ca/3955185 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-12-06-02:05:41/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Wed Dec 6 02:04:35 2017
 UPTIME: 15 days, 00:23:09
LOAD AVERAGE: 6.23, 4.95, 3.77
 TASKS: 726
 NODENAME: server
 RELEASE: 3.10.0-693.5.2.el7.x86_64
 VERSION: #1 SMP Fri Oct 20 20:32:50 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 00000000000008d0"


bt output



PID: 31570 TASK: ffff8811f9f9dee0 CPU: 11 COMMAND: "mysqld"
 #0 [ffff8800988ff340] machine_kexec at ffffffff8105c4cb
 #1 [ffff8800988ff3a0] __crash_kexec at ffffffff81104a42
 #2 [ffff8800988ff470] crash_kexec at ffffffff81104b30
 #3 [ffff8800988ff488] oops_end at ffffffff816ad338
 #4 [ffff8800988ff4b0] no_context at ffffffff8169d35a
 #5 [ffff8800988ff500] __bad_area_nosemaphore at ffffffff8169d3f0
 #6 [ffff8800988ff548] bad_area at ffffffff8169d714
 #7 [ffff8800988ff570] __do_page_fault at ffffffff816b02fc
 #8 [ffff8800988ff5d0] do_page_fault at ffffffff816b03a5
 #9 [ffff8800988ff600] page_fault at ffffffff816ac5c8
 [exception RIP: xfs_fs_destroy_inode+78]
 RIP: ffffffffc03d6f3e RSP: ffff8800988ff6b0 RFLAGS: 00010202
 RAX: 0000000000000000 RBX: ffff8809781d00f4 RCX: 0000000000020100
 RDX: 000000000000000b RSI: ffff8809781d01d8 RDI: ffff8809781d0000
 RBP: ffff8800988ff6c8 R8: 0000000000000000 R9: 09781d01f00c0000
 R10: f669e55c17347c03 R11: 0000000000000000 R12: ffff8809781d0000
 R13: ffff8809781d0150 R14: ffff8811f97c2108 R15: ffff880cd7cbfe08
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff8800988ff6d0] evict at ffffffff8121d658
#11 [ffff8800988ff710] iget_locked at ffffffff8121d83e
#12 [ffff8800988ff738] prune_icache_sb at ffffffff8121e834
#13 [ffff8800988ff7f0] vmpressure at ffffffff811f7451
#14 [ffff8800988ff878] do_try_to_free_pages at ffffffff811985a2
#15 [ffff8800988ff8f0] try_to_free_pages at ffffffff811987bc
#16 [ffff8800988ff988] __alloc_pages_slowpath at ffffffff8169fbcb
#17 [ffff8800988ffa78] __alloc_pages_nodemask at ffffffff8118cdb5
#18 [ffff8800988ffb28] alloc_pages_current at ffffffff811d1078
#19 [ffff8800988ffb70] alloc_skb_with_frags at ffffffff81573cbd
#20 [ffff8800988ffbc0] sock_alloc_send_pskb at ffffffff8156c9b9
#21 [ffff8800988ffc48] unix_stream_sendmsg at ffffffff8163aef0
#22 [ffff8800988ffcf8] sock_sendmsg at ffffffff8156a580
#23 [ffff8800988ffe58] SYSC_sendto at ffffffff8156a731
#24 [ffff8800988fff70] sys_sendto at ffffffff8156b2ce
#25 [ffff8800988fff80] system_call_fastpath at ffffffff816b5089
 RIP: 00007fca94f07c0b RSP: 00007fca7fed7308 RFLAGS: 00000202
 RAX: 000000000000002c RBX: ffffffff816b5089 RCX: 00000000014c8a7c
 RDX: 0000000000004000 RSI: 00007fca6ebdb008 RDI: 0000000000000163
 RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000000
 R10: 0000000000000040 R11: 0000000000000246 R12: ffffffff8156b2ce
 R13: ffff8800988fff78 R14: 00007fca6ebef238 R15: 0000000000004000
 ORIG_RAX: 000000000000002c CS: 0033 SS: 002b


https://pastebin.ca/3955186 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-14-04:21:30/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Tue Nov 14 04:20:23 2017
 UPTIME: 09:25:26
LOAD AVERAGE: 2.21, 1.47, 1.71
 TASKS: 788
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008"


bt output



PID: 29737 TASK: ffff8811f6346eb0 CPU: 3 COMMAND: "kworker/3:0"
 #0 [ffff8811ac617948] machine_kexec at ffffffff8105c4cb
 #1 [ffff8811ac6179a8] __crash_kexec at ffffffff81104a32
 #2 [ffff8811ac617a78] crash_kexec at ffffffff81104b20
 #3 [ffff8811ac617ab8] no_context at ffffffff8169d2ba
 #4 [ffff8811ac617b08] no_context at ffffffff8169d350
 #5 [ffff8811ac617b50] __bad_area_nosemaphore at ffffffff8169d4ba
 #6 [ffff8811ac617b60] __do_page_fault at ffffffff816b017e
 #7 [ffff8811ac617bc0] __do_page_fault at ffffffff816b0325
 #8 [ffff8811ac617bf0] general_protection at ffffffff816ac548
 [exception RIP: xlog_write+772]
 RIP: ffffffffc03f8644 RSP: ffff8811ac617ca0 RFLAGS: 00010286
 RAX: 0000000000000000 RBX: ffff8811af615870 RCX: 0000000000000000
 RDX: 00000000000261a0 RSI: 0000000000000000 RDI: 0000000000005e6c
 RBP: ffff8811ac617d38 R8: 0000000000000000 R9: ffffc9000d576078
 R10: ffffc9000d57606c R11: ffff8808f53ad800 R12: 0000000000007e00
 R13: 00000000000000d0 R14: 0000000000000000 R15: ffff880c9c7bb000
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff8811ac617d40] xlog_cil_push at ffffffffc03fa1f8 [xfs]
#10 [ffff8811ac617e10] xlog_cil_push_work at ffffffffc03fa395 [xfs]
#11 [ffff8811ac617e20] process_one_work at ffffffff810a881a
#12 [ffff8811ac617e68] worker_thread at ffffffff810a94e6
#13 [ffff8811ac617ec8] kthread at ffffffff810b098f
#14 [ffff8811ac617f50] save_rest at ffffffff816b4f58


https://pastebin.ca/3955187 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-13-18:50:12/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Mon Nov 13 18:49:05 2017
 UPTIME: 10 days, 17:30:03
LOAD AVERAGE: 2.10, 2.43, 2.33
 TASKS: 785
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000020"


btoutput



PID: 27951 TASK: ffff8808f1b25ee0 CPU: 16 COMMAND: "php-cgi"
 #0 [ffff880113643a90] machine_kexec at ffffffff8105c4cb
 #1 [ffff880113643af0] __crash_kexec at ffffffff81104a32
 #2 [ffff880113643bc0] crash_kexec at ffffffff81104b20
 #3 [ffff880113643c00] no_context at ffffffff8169d2ba
 #4 [ffff880113643c50] no_context at ffffffff8169d350
 #5 [ffff880113643c98] mm_fault_error at ffffffff8169d674
 #6 [ffff880113643cc0] __do_page_fault at ffffffff816b027c
 #7 [ffff880113643d20] __do_page_fault at ffffffff816b0325
 #8 [ffff880113643d50] general_protection at ffffffff816ac548
 [exception RIP: xfs_free_eofblocks+91]
 RIP: ffffffffc03cf8db RSP: ffff880113643e00 RFLAGS: 00010212
 RAX: 0000000000000000 RBX: ffff8811f3bdcec0 RCX: 000000000000000c
 RDX: 0000000000000fff RSI: 0000000000000001 RDI: ffff8811f3bdcec0
 RBP: ffff880113643e58 R8: 0000000000000000 R9: 0000000000000000
 R10: ffff8811f3bdd010 R11: ffff880017c7b710 R12: 0000000000015a0b
 R13: ffff8808f7346000 R14: ffff8808f4ba00c0 R15: ffff880170ad2d20
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff880113643e60] xfs_release at ffffffffc03e7105 [xfs]
#10 [ffff880113643e88] xfs_file_release at ffffffffc03d65a5 [xfs]
#11 [ffff880113643e98] delayed_fput at ffffffff81202fc9
#12 [ffff880113643ee0] alloc_file at ffffffff8120322e
#13 [ffff880113643ef0] task_work_run at ffffffff810ad247
#14 [ffff880113643f30] do_notify_resume at ffffffff8102ab62
#15 [ffff880113643f50] int_with_check at ffffffff816b52bd
 RIP: 00007fd2f72abe90 RSP: 00007ffe5d25d848 RFLAGS: 00000246
 RAX: 0000000000000000 RBX: 000055bac5cda3c0 RCX: ffffffffffffffff
 RDX: 0000000000000008 RSI: 0000000000000001 RDI: 0000000000000003
 RBP: 0000000000000000 R8: 000055bac5cda4a0 R9: 00007fd2fa6bb840
 R10: 000000000000000d R11: 0000000000000246 R12: 0000000000000000
 R13: 0000000000000001 R14: 00007fd2e8771f58 R15: 0000000000000000
 ORIG_RAX: 0000000000000003 CS: 0033 SS: 002b


https://pastebin.ca/3955188 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-03-01:14:25/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Fri Nov 3 01:13:19 2017
 UPTIME: 3 days, 18:53:59
LOAD AVERAGE: 2.66, 2.58, 2.16
 TASKS: 744
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at (null)"


bt output



PID: 144 TASK: ffff8808faf52f70 CPU: 7 COMMAND: "kswapd1"
 #0 [ffff8808f9e0f7e8] machine_kexec at ffffffff8105c4cb
 #1 [ffff8808f9e0f848] __crash_kexec at ffffffff81104a32
 #2 [ffff8808f9e0f918] crash_kexec at ffffffff81104b20
 #3 [ffff8808f9e0f958] no_context at ffffffff8169d2ba
 #4 [ffff8808f9e0f9a8] no_context at ffffffff8169d350
 #5 [ffff8808f9e0f9f0] __bad_area_nosemaphore at ffffffff8169d4ba
 #6 [ffff8808f9e0fa00] __do_page_fault at ffffffff816b017e
 #7 [ffff8808f9e0fa08] __radix_tree_create at ffffffff81328a9e
 #8 [ffff8808f9e0fa60] __do_page_fault at ffffffff816b0325
 #9 [ffff8808f9e0fa90] general_protection at ffffffff816ac548
 [exception RIP: crc32_generic_combine+89]
 RIP: ffffffff8133db39 RSP: ffff8808f9e0fb40 RFLAGS: 00010207
 RAX: 0000000000000000 RBX: ffff88010b138008 RCX: dead000000000200
 RDX: 0000000000000000 RSI: ffff88010b8bc570 RDI: ffff88010b138008
 RBP: ffff8808f9e0fb40 R8: e018000000000000 R9: 010b8bc5700c0000
 R10: fed6747d72e55c03 R11: 0000000000000000 R12: ffff88010b138000
 R13: ffff8808f47dbcb8 R14: ffff8811f973f108 R15: ffff880143a76b48
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#10 [ffff8808f9e0fb48] file_has_perm at ffffffff812b752d
#11 [ffff8808f9e0fb88] evict at ffffffff8121d5e2
#12 [ffff8808f9e0fbb8] iget_locked at ffffffff8121d81a
#13 [ffff8808f9e0fbe0] iget_locked at ffffffff8121d8ce
#14 [ffff8808f9e0fc08] new_inode_pseudo at ffffffff8121e8c4
#15 [ffff8808f9e0fc70] ns_set_super at ffffffff81203878
#16 [ffff8808f9e0fca8] shrink_slab at ffffffff81195413
#17 [ffff8808f9e0fcc0] vmpressure_register_event at ffffffff811f7547
#18 [ffff8808f9e0fd48] balance_pgdat at ffffffff81199081
#19 [ffff8808f9e0fe20] kswapd at ffffffff81199323
#20 [ffff8808f9e0fe78] wake_up_atomic_t at ffffffff810b1910
#21 [ffff8808f9e0fea8] balance_pgdat at ffffffff811991b0
#22 [ffff8808f9e0fec8] kthread at ffffffff810b098f
#23 [ffff8808f9e0ff50] save_rest at ffffffff816b4f58

https://pastebin.ca/3955189

 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-10-30-07:14:38/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Mon Oct 30 06:13:31 2017
 UPTIME: 3 days, 06:21:18
LOAD AVERAGE: 1.03, 1.19, 1.36
 TASKS: 707
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "general protection fault: 0000 [#1] SMP "


bt output



PID: 32005 TASK: ffff8808f8bfeeb0 CPU: 4 COMMAND: "vtund"
 #0 [ffff8804623fb950] machine_kexec at ffffffff8105c4cb
 #1 [ffff8804623fb9b0] __crash_kexec at ffffffff81104a32
 #2 [ffff8804623fba80] crash_kexec at ffffffff81104b20
 #3 [ffff8804623fbac0] die at ffffffff8102e97b
 #4 [ffff8804623fbaf0] do_general_protection at ffffffff816acc3e
 #5 [ffff8804623fbb20] xen_int3 at ffffffff816ac4e8
 [exception RIP: memcmp]
 RIP: ffffffff8132b980 RSP: ffff8804623fbbd0 RFLAGS: 00010286
 RAX: ffff8801f3cd7048 RBX: 000000000000000a RCX: ffff8806917dc048
 RDX: ffff8808fb333800 RSI: ffff8808f9f482e8 RDI: 0236303633043a00
 RBP: ffff8804623fbc40 R8: 0000000000000036 R9: 0000000000000000
 R10: ffff88017fc03400 R11: 0000000000000000 R12: ffffffff8197ea30
 R13: 0236303633043a00 R14: ffff8808fb333868 R15: ffff8808a4f6a300
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
 #6 [ffff8804623fbbd0] insert_header at ffffffff8127cb99
 #7 [ffff8804623fbc48] register_leaf_sysctl_tables at ffffffff8127d28b
 #8 [ffff8804623fbc60] bstr_printf at ffffffff8132fb59
 #9 [ffff8804623fbd40] ipv6_add_dev at ffffffff8164784a
#10 [ffff8804623fbd70] addrconf_notify at ffffffff8164c9c9
#11 [ffff8804623fbd78] sysfs_slab_alias at ffffffff811df8d6
#12 [ffff8804623fbd88] dropmon_net_event at ffffffff815ab686
#13 [ffff8804623fbde0] trace_do_page_fault at ffffffff816b051c
#14 [ffff8804623fbe18] raw_notifier_call_chain at ffffffff810b68d6
#15 [ffff8804623fbe28] call_netdevice_notifiers_info at ffffffff8158306d
#16 [ffff8804623fbe50] register_netdevice at ffffffff8158c576
#17 [ffff8804623fbe88] __tun_chr_ioctl at ffffffffc07fca8e [tun]
#18 [ffff8804623fbf20] tun_chr_compat_ioctl at ffffffffc07fd10b [tun]
#19 [ffff8804623fbf30] compat_sys_ioctl at ffffffff8125d73b
#20 [ffff8804623fbf80] cstar_tracesys at ffffffff816b746c
 RIP: 00000000080d458c RSP: 00000000fff1980c RFLAGS: 00000246
 RAX: ffffffffffffffda RBX: ffffffff816b746c RCX: 00000000400454ca
 RDX: 00000000fff19840 RSI: 00000000fff19840 RDI: 00000000fff19860
 RBP: 00000000fff19878 R8: 0000000000000000 R9: 0000000000000000
 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 ORIG_RAX: 0000000000000036 CS: 0023 SS: 002b



centos kernel-panic




share|improve this question






















  • I suggest you directly put all the relevant information here. With proper formatting you can show all of them in a ordered manner without worrying that it would distract people from your main question.
    – Weijun Zhou
    Jan 4 at 3:25












up vote
1
down vote

favorite
1









up vote
1
down vote

favorite
1






1





I have a HP DL360 G7 running CentOS Linux release 7.4.1708 (Core) and suffering from kernel panics at random intervals and (to my eyes) random reasons. I've dug into the crash dumps using these as a guides: https://www.slideshare.net/PaulVNovarese/linux-crash-dump-capture-and-analysis and https://www.dedoimedo.com/computers/crash-analyze.html



But the crash dumps seem to my eyes pointing to a different reason each time, i.e, I don't have the competency to understand crash dumps well enough to determine if they have any connection.



All the crash dumps that I have managed to get. Paste contains crash commands sys, bt and log of each dump.



EDIT

Due to suggested edits, I've pasted the sys and bt outputs of crash command. I'll keep the pastebin links as the log output of crash is too large to fit here.



Also adding that the server is running ECC memory and no indication of MCEs appear in logs so I'm not convinced that this is a faulty memory situation.



Any help or guide how to solve this would be greatly appreciated.



https://pastebin.ca/3955184



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2018-01-03-03:08:48/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Wed Jan 3 03:07:40 2018
 UPTIME: 28 days, 00:57:45
LOAD AVERAGE: 3.45, 2.43, 2.66
 TASKS: 714
 NODENAME: server
 RELEASE: 3.10.0-693.11.1.el7.x86_64
 VERSION: #1 SMP Mon Dec 4 23:52:40 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "general protection fault: 0000 [#1] SMP "


btoutput



PID: 24892 TASK: ffff8808f9111fa0 CPU: 0 COMMAND: "python"
 #0 [ffff8808fba03910] machine_kexec at ffffffff8105c52b
 #1 [ffff8808fba03970] __crash_kexec at ffffffff81104a42
 #2 [ffff8808fba03a40] crash_kexec at ffffffff81104b30
 #3 [ffff8808fba03a58] oops_end at ffffffff816ad338
 #4 [ffff8808fba03a80] die at ffffffff8102e97b
 #5 [ffff8808fba03ab0] do_general_protection at ffffffff816accbe
 #6 [ffff8808fba03ae0] general_protection at ffffffff816ac568
 [exception RIP: inet6_csk_search_req+261]
 RIP: ffffffff81673385 RSP: ffff8808fba03b98 RFLAGS: 00010202
 RAX: 0000000000001c9e RBX: 3932383931333431 RCX: ffff8807ed7918c8
 RDX: 00000000ffffffff RSI: 0000000000000000 RDI: 00000000b1fa562e
 RBP: ffff8808fba03bb0 R8: ffff8807ed7918d8 R9: 0000000000000001
 R10: ffff8801f3ffc000 R11: 00000000b80dcf9c R12: ffff880783d27178
 R13: ffff8808fba03bd0 R14: ffff8808ec21a9a8 R15: 0000000000000000
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #7 [ffff8808fba03bb8] tcp_v6_do_rcv at ffffffff8166d73a
 #8 [ffff8808fba03c10] tcp_v6_rcv at ffffffff8166e1b2
 #9 [ffff8808fba03cc0] ip6_input_finish at ffffffff81643712
#10 [ffff8808fba03d00] ip6_input at ffffffff81643fe3
#11 [ffff8808fba03d58] ip6_rcv_finish at ffffffff81643518
#12 [ffff8808fba03d70] ipv6_rcv at ffffffff81643d99
#13 [ffff8808fba03df0] __netif_receive_skb_core at ffffffff81586f22
#14 [ffff8808fba03e60] __netif_receive_skb at ffffffff81587188
#15 [ffff8808fba03e80] process_backlog at ffffffff8158841e
#16 [ffff8808fba03ec0] net_rx_action at ffffffff8158799d
#17 [ffff8808fba03f40] __do_softirq at ffffffff81090b4f
#18 [ffff8808fba03fb0] call_softirq at ffffffff816b6b1c
--- <IRQ stack> ---
#19 [ffff880c1fcb3980] local_bh_enable at ffffffff81090017
#20 [ffff880c1fcb3990] __dev_queue_xmit at ffffffff815895a5
#21 [ffff880c1fcb39e8] local_bh_enable at ffffffff81090017
#22 [ffff880c1fcb39f8] ip6_finish_output2 at ffffffff816407b1
#23 [ffff880c1fcb3a78] ip6_finish_output at ffffffff81642cbc
#24 [ffff880c1fcb3aa0] ip6_output at ffffffff81642d77
#25 [ffff880c1fcb3b00] ip6_xmit at ffffffff81640039
#26 [ffff880c1fcb3ba8] inet6_csk_xmit at ffffffff81673059
#27 [ffff880c1fcb3c48] tcp_transmit_skb at ffffffff815e7c9f
#28 [ffff880c1fcb3cb8] tcp_connect at ffffffff815e97ed
#29 [ffff880c1fcb3d38] tcp_v6_connect at ffffffff8166c106
#30 [ffff880c1fcb3e08] __inet_stream_connect at ffffffff81605725
#31 [ffff880c1fcb3e80] inet_stream_connect at ffffffff816059d8
#32 [ffff880c1fcb3eb0] SYSC_connect at ffffffff8156a497
#33 [ffff880c1fcb3f70] sys_connect at ffffffff8156b29e
#34 [ffff880c1fcb3f80] system_call_fastpath at ffffffff816b5089
 RIP: 00007f9e58f319d0 RSP: 00007ffece4029e0 RFLAGS: 00010246
 RAX: 000000000000002a RBX: ffffffff816b5089 RCX: 00007f9e59144c28
 RDX: 000000000000001c RSI: 00007ffece4030d0 RDI: 0000000000000003
 RBP: 0000000000000000 R8: 0000000000000000 R9: 00000000032dfb60
 R10: 0000000000000006 R11: 0000000000000246 R12: ffffffff8156b29e
 R13: ffff880c1fcb3f78 R14: 0000000002b56510 R15: 00000000025950a0
 ORIG_RAX: 000000000000002a CS: 0033 SS: 002b


https://pastebin.ca/3955185 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-12-06-02:05:41/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Wed Dec 6 02:04:35 2017
 UPTIME: 15 days, 00:23:09
LOAD AVERAGE: 6.23, 4.95, 3.77
 TASKS: 726
 NODENAME: server
 RELEASE: 3.10.0-693.5.2.el7.x86_64
 VERSION: #1 SMP Fri Oct 20 20:32:50 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 00000000000008d0"


bt output



PID: 31570 TASK: ffff8811f9f9dee0 CPU: 11 COMMAND: "mysqld"
 #0 [ffff8800988ff340] machine_kexec at ffffffff8105c4cb
 #1 [ffff8800988ff3a0] __crash_kexec at ffffffff81104a42
 #2 [ffff8800988ff470] crash_kexec at ffffffff81104b30
 #3 [ffff8800988ff488] oops_end at ffffffff816ad338
 #4 [ffff8800988ff4b0] no_context at ffffffff8169d35a
 #5 [ffff8800988ff500] __bad_area_nosemaphore at ffffffff8169d3f0
 #6 [ffff8800988ff548] bad_area at ffffffff8169d714
 #7 [ffff8800988ff570] __do_page_fault at ffffffff816b02fc
 #8 [ffff8800988ff5d0] do_page_fault at ffffffff816b03a5
 #9 [ffff8800988ff600] page_fault at ffffffff816ac5c8
 [exception RIP: xfs_fs_destroy_inode+78]
 RIP: ffffffffc03d6f3e RSP: ffff8800988ff6b0 RFLAGS: 00010202
 RAX: 0000000000000000 RBX: ffff8809781d00f4 RCX: 0000000000020100
 RDX: 000000000000000b RSI: ffff8809781d01d8 RDI: ffff8809781d0000
 RBP: ffff8800988ff6c8 R8: 0000000000000000 R9: 09781d01f00c0000
 R10: f669e55c17347c03 R11: 0000000000000000 R12: ffff8809781d0000
 R13: ffff8809781d0150 R14: ffff8811f97c2108 R15: ffff880cd7cbfe08
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff8800988ff6d0] evict at ffffffff8121d658
#11 [ffff8800988ff710] iget_locked at ffffffff8121d83e
#12 [ffff8800988ff738] prune_icache_sb at ffffffff8121e834
#13 [ffff8800988ff7f0] vmpressure at ffffffff811f7451
#14 [ffff8800988ff878] do_try_to_free_pages at ffffffff811985a2
#15 [ffff8800988ff8f0] try_to_free_pages at ffffffff811987bc
#16 [ffff8800988ff988] __alloc_pages_slowpath at ffffffff8169fbcb
#17 [ffff8800988ffa78] __alloc_pages_nodemask at ffffffff8118cdb5
#18 [ffff8800988ffb28] alloc_pages_current at ffffffff811d1078
#19 [ffff8800988ffb70] alloc_skb_with_frags at ffffffff81573cbd
#20 [ffff8800988ffbc0] sock_alloc_send_pskb at ffffffff8156c9b9
#21 [ffff8800988ffc48] unix_stream_sendmsg at ffffffff8163aef0
#22 [ffff8800988ffcf8] sock_sendmsg at ffffffff8156a580
#23 [ffff8800988ffe58] SYSC_sendto at ffffffff8156a731
#24 [ffff8800988fff70] sys_sendto at ffffffff8156b2ce
#25 [ffff8800988fff80] system_call_fastpath at ffffffff816b5089
 RIP: 00007fca94f07c0b RSP: 00007fca7fed7308 RFLAGS: 00000202
 RAX: 000000000000002c RBX: ffffffff816b5089 RCX: 00000000014c8a7c
 RDX: 0000000000004000 RSI: 00007fca6ebdb008 RDI: 0000000000000163
 RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000000
 R10: 0000000000000040 R11: 0000000000000246 R12: ffffffff8156b2ce
 R13: ffff8800988fff78 R14: 00007fca6ebef238 R15: 0000000000004000
 ORIG_RAX: 000000000000002c CS: 0033 SS: 002b


https://pastebin.ca/3955186 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-14-04:21:30/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Tue Nov 14 04:20:23 2017
 UPTIME: 09:25:26
LOAD AVERAGE: 2.21, 1.47, 1.71
 TASKS: 788
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008"


bt output



PID: 29737 TASK: ffff8811f6346eb0 CPU: 3 COMMAND: "kworker/3:0"
 #0 [ffff8811ac617948] machine_kexec at ffffffff8105c4cb
 #1 [ffff8811ac6179a8] __crash_kexec at ffffffff81104a32
 #2 [ffff8811ac617a78] crash_kexec at ffffffff81104b20
 #3 [ffff8811ac617ab8] no_context at ffffffff8169d2ba
 #4 [ffff8811ac617b08] no_context at ffffffff8169d350
 #5 [ffff8811ac617b50] __bad_area_nosemaphore at ffffffff8169d4ba
 #6 [ffff8811ac617b60] __do_page_fault at ffffffff816b017e
 #7 [ffff8811ac617bc0] __do_page_fault at ffffffff816b0325
 #8 [ffff8811ac617bf0] general_protection at ffffffff816ac548
 [exception RIP: xlog_write+772]
 RIP: ffffffffc03f8644 RSP: ffff8811ac617ca0 RFLAGS: 00010286
 RAX: 0000000000000000 RBX: ffff8811af615870 RCX: 0000000000000000
 RDX: 00000000000261a0 RSI: 0000000000000000 RDI: 0000000000005e6c
 RBP: ffff8811ac617d38 R8: 0000000000000000 R9: ffffc9000d576078
 R10: ffffc9000d57606c R11: ffff8808f53ad800 R12: 0000000000007e00
 R13: 00000000000000d0 R14: 0000000000000000 R15: ffff880c9c7bb000
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff8811ac617d40] xlog_cil_push at ffffffffc03fa1f8 [xfs]
#10 [ffff8811ac617e10] xlog_cil_push_work at ffffffffc03fa395 [xfs]
#11 [ffff8811ac617e20] process_one_work at ffffffff810a881a
#12 [ffff8811ac617e68] worker_thread at ffffffff810a94e6
#13 [ffff8811ac617ec8] kthread at ffffffff810b098f
#14 [ffff8811ac617f50] save_rest at ffffffff816b4f58


https://pastebin.ca/3955187 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-13-18:50:12/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Mon Nov 13 18:49:05 2017
 UPTIME: 10 days, 17:30:03
LOAD AVERAGE: 2.10, 2.43, 2.33
 TASKS: 785
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000020"


btoutput



PID: 27951 TASK: ffff8808f1b25ee0 CPU: 16 COMMAND: "php-cgi"
 #0 [ffff880113643a90] machine_kexec at ffffffff8105c4cb
 #1 [ffff880113643af0] __crash_kexec at ffffffff81104a32
 #2 [ffff880113643bc0] crash_kexec at ffffffff81104b20
 #3 [ffff880113643c00] no_context at ffffffff8169d2ba
 #4 [ffff880113643c50] no_context at ffffffff8169d350
 #5 [ffff880113643c98] mm_fault_error at ffffffff8169d674
 #6 [ffff880113643cc0] __do_page_fault at ffffffff816b027c
 #7 [ffff880113643d20] __do_page_fault at ffffffff816b0325
 #8 [ffff880113643d50] general_protection at ffffffff816ac548
 [exception RIP: xfs_free_eofblocks+91]
 RIP: ffffffffc03cf8db RSP: ffff880113643e00 RFLAGS: 00010212
 RAX: 0000000000000000 RBX: ffff8811f3bdcec0 RCX: 000000000000000c
 RDX: 0000000000000fff RSI: 0000000000000001 RDI: ffff8811f3bdcec0
 RBP: ffff880113643e58 R8: 0000000000000000 R9: 0000000000000000
 R10: ffff8811f3bdd010 R11: ffff880017c7b710 R12: 0000000000015a0b
 R13: ffff8808f7346000 R14: ffff8808f4ba00c0 R15: ffff880170ad2d20
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff880113643e60] xfs_release at ffffffffc03e7105 [xfs]
#10 [ffff880113643e88] xfs_file_release at ffffffffc03d65a5 [xfs]
#11 [ffff880113643e98] delayed_fput at ffffffff81202fc9
#12 [ffff880113643ee0] alloc_file at ffffffff8120322e
#13 [ffff880113643ef0] task_work_run at ffffffff810ad247
#14 [ffff880113643f30] do_notify_resume at ffffffff8102ab62
#15 [ffff880113643f50] int_with_check at ffffffff816b52bd
 RIP: 00007fd2f72abe90 RSP: 00007ffe5d25d848 RFLAGS: 00000246
 RAX: 0000000000000000 RBX: 000055bac5cda3c0 RCX: ffffffffffffffff
 RDX: 0000000000000008 RSI: 0000000000000001 RDI: 0000000000000003
 RBP: 0000000000000000 R8: 000055bac5cda4a0 R9: 00007fd2fa6bb840
 R10: 000000000000000d R11: 0000000000000246 R12: 0000000000000000
 R13: 0000000000000001 R14: 00007fd2e8771f58 R15: 0000000000000000
 ORIG_RAX: 0000000000000003 CS: 0033 SS: 002b


https://pastebin.ca/3955188 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-03-01:14:25/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Fri Nov 3 01:13:19 2017
 UPTIME: 3 days, 18:53:59
LOAD AVERAGE: 2.66, 2.58, 2.16
 TASKS: 744
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at (null)"


bt output



PID: 144 TASK: ffff8808faf52f70 CPU: 7 COMMAND: "kswapd1"
 #0 [ffff8808f9e0f7e8] machine_kexec at ffffffff8105c4cb
 #1 [ffff8808f9e0f848] __crash_kexec at ffffffff81104a32
 #2 [ffff8808f9e0f918] crash_kexec at ffffffff81104b20
 #3 [ffff8808f9e0f958] no_context at ffffffff8169d2ba
 #4 [ffff8808f9e0f9a8] no_context at ffffffff8169d350
 #5 [ffff8808f9e0f9f0] __bad_area_nosemaphore at ffffffff8169d4ba
 #6 [ffff8808f9e0fa00] __do_page_fault at ffffffff816b017e
 #7 [ffff8808f9e0fa08] __radix_tree_create at ffffffff81328a9e
 #8 [ffff8808f9e0fa60] __do_page_fault at ffffffff816b0325
 #9 [ffff8808f9e0fa90] general_protection at ffffffff816ac548
 [exception RIP: crc32_generic_combine+89]
 RIP: ffffffff8133db39 RSP: ffff8808f9e0fb40 RFLAGS: 00010207
 RAX: 0000000000000000 RBX: ffff88010b138008 RCX: dead000000000200
 RDX: 0000000000000000 RSI: ffff88010b8bc570 RDI: ffff88010b138008
 RBP: ffff8808f9e0fb40 R8: e018000000000000 R9: 010b8bc5700c0000
 R10: fed6747d72e55c03 R11: 0000000000000000 R12: ffff88010b138000
 R13: ffff8808f47dbcb8 R14: ffff8811f973f108 R15: ffff880143a76b48
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#10 [ffff8808f9e0fb48] file_has_perm at ffffffff812b752d
#11 [ffff8808f9e0fb88] evict at ffffffff8121d5e2
#12 [ffff8808f9e0fbb8] iget_locked at ffffffff8121d81a
#13 [ffff8808f9e0fbe0] iget_locked at ffffffff8121d8ce
#14 [ffff8808f9e0fc08] new_inode_pseudo at ffffffff8121e8c4
#15 [ffff8808f9e0fc70] ns_set_super at ffffffff81203878
#16 [ffff8808f9e0fca8] shrink_slab at ffffffff81195413
#17 [ffff8808f9e0fcc0] vmpressure_register_event at ffffffff811f7547
#18 [ffff8808f9e0fd48] balance_pgdat at ffffffff81199081
#19 [ffff8808f9e0fe20] kswapd at ffffffff81199323
#20 [ffff8808f9e0fe78] wake_up_atomic_t at ffffffff810b1910
#21 [ffff8808f9e0fea8] balance_pgdat at ffffffff811991b0
#22 [ffff8808f9e0fec8] kthread at ffffffff810b098f
#23 [ffff8808f9e0ff50] save_rest at ffffffff816b4f58

https://pastebin.ca/3955189

 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-10-30-07:14:38/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Mon Oct 30 06:13:31 2017
 UPTIME: 3 days, 06:21:18
LOAD AVERAGE: 1.03, 1.19, 1.36
 TASKS: 707
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "general protection fault: 0000 [#1] SMP "


bt output



PID: 32005 TASK: ffff8808f8bfeeb0 CPU: 4 COMMAND: "vtund"
 #0 [ffff8804623fb950] machine_kexec at ffffffff8105c4cb
 #1 [ffff8804623fb9b0] __crash_kexec at ffffffff81104a32
 #2 [ffff8804623fba80] crash_kexec at ffffffff81104b20
 #3 [ffff8804623fbac0] die at ffffffff8102e97b
 #4 [ffff8804623fbaf0] do_general_protection at ffffffff816acc3e
 #5 [ffff8804623fbb20] xen_int3 at ffffffff816ac4e8
 [exception RIP: memcmp]
 RIP: ffffffff8132b980 RSP: ffff8804623fbbd0 RFLAGS: 00010286
 RAX: ffff8801f3cd7048 RBX: 000000000000000a RCX: ffff8806917dc048
 RDX: ffff8808fb333800 RSI: ffff8808f9f482e8 RDI: 0236303633043a00
 RBP: ffff8804623fbc40 R8: 0000000000000036 R9: 0000000000000000
 R10: ffff88017fc03400 R11: 0000000000000000 R12: ffffffff8197ea30
 R13: 0236303633043a00 R14: ffff8808fb333868 R15: ffff8808a4f6a300
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
 #6 [ffff8804623fbbd0] insert_header at ffffffff8127cb99
 #7 [ffff8804623fbc48] register_leaf_sysctl_tables at ffffffff8127d28b
 #8 [ffff8804623fbc60] bstr_printf at ffffffff8132fb59
 #9 [ffff8804623fbd40] ipv6_add_dev at ffffffff8164784a
#10 [ffff8804623fbd70] addrconf_notify at ffffffff8164c9c9
#11 [ffff8804623fbd78] sysfs_slab_alias at ffffffff811df8d6
#12 [ffff8804623fbd88] dropmon_net_event at ffffffff815ab686
#13 [ffff8804623fbde0] trace_do_page_fault at ffffffff816b051c
#14 [ffff8804623fbe18] raw_notifier_call_chain at ffffffff810b68d6
#15 [ffff8804623fbe28] call_netdevice_notifiers_info at ffffffff8158306d
#16 [ffff8804623fbe50] register_netdevice at ffffffff8158c576
#17 [ffff8804623fbe88] __tun_chr_ioctl at ffffffffc07fca8e [tun]
#18 [ffff8804623fbf20] tun_chr_compat_ioctl at ffffffffc07fd10b [tun]
#19 [ffff8804623fbf30] compat_sys_ioctl at ffffffff8125d73b
#20 [ffff8804623fbf80] cstar_tracesys at ffffffff816b746c
 RIP: 00000000080d458c RSP: 00000000fff1980c RFLAGS: 00000246
 RAX: ffffffffffffffda RBX: ffffffff816b746c RCX: 00000000400454ca
 RDX: 00000000fff19840 RSI: 00000000fff19840 RDI: 00000000fff19860
 RBP: 00000000fff19878 R8: 0000000000000000 R9: 0000000000000000
 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 ORIG_RAX: 0000000000000036 CS: 0023 SS: 002b



centos kernel-panic




share|improve this question














I have a HP DL360 G7 running CentOS Linux release 7.4.1708 (Core) and suffering from kernel panics at random intervals and (to my eyes) random reasons. I've dug into the crash dumps using these as a guides: https://www.slideshare.net/PaulVNovarese/linux-crash-dump-capture-and-analysis and https://www.dedoimedo.com/computers/crash-analyze.html



But the crash dumps seem to my eyes pointing to a different reason each time, i.e, I don't have the competency to understand crash dumps well enough to determine if they have any connection.



All the crash dumps that I have managed to get. Paste contains crash commands sys, bt and log of each dump.



EDIT

Due to suggested edits, I've pasted the sys and bt outputs of crash command. I'll keep the pastebin links as the log output of crash is too large to fit here.



Also adding that the server is running ECC memory and no indication of MCEs appear in logs so I'm not convinced that this is a faulty memory situation.



Any help or guide how to solve this would be greatly appreciated.



https://pastebin.ca/3955184



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2018-01-03-03:08:48/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Wed Jan 3 03:07:40 2018
 UPTIME: 28 days, 00:57:45
LOAD AVERAGE: 3.45, 2.43, 2.66
 TASKS: 714
 NODENAME: server
 RELEASE: 3.10.0-693.11.1.el7.x86_64
 VERSION: #1 SMP Mon Dec 4 23:52:40 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "general protection fault: 0000 [#1] SMP "


btoutput



PID: 24892 TASK: ffff8808f9111fa0 CPU: 0 COMMAND: "python"
 #0 [ffff8808fba03910] machine_kexec at ffffffff8105c52b
 #1 [ffff8808fba03970] __crash_kexec at ffffffff81104a42
 #2 [ffff8808fba03a40] crash_kexec at ffffffff81104b30
 #3 [ffff8808fba03a58] oops_end at ffffffff816ad338
 #4 [ffff8808fba03a80] die at ffffffff8102e97b
 #5 [ffff8808fba03ab0] do_general_protection at ffffffff816accbe
 #6 [ffff8808fba03ae0] general_protection at ffffffff816ac568
 [exception RIP: inet6_csk_search_req+261]
 RIP: ffffffff81673385 RSP: ffff8808fba03b98 RFLAGS: 00010202
 RAX: 0000000000001c9e RBX: 3932383931333431 RCX: ffff8807ed7918c8
 RDX: 00000000ffffffff RSI: 0000000000000000 RDI: 00000000b1fa562e
 RBP: ffff8808fba03bb0 R8: ffff8807ed7918d8 R9: 0000000000000001
 R10: ffff8801f3ffc000 R11: 00000000b80dcf9c R12: ffff880783d27178
 R13: ffff8808fba03bd0 R14: ffff8808ec21a9a8 R15: 0000000000000000
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #7 [ffff8808fba03bb8] tcp_v6_do_rcv at ffffffff8166d73a
 #8 [ffff8808fba03c10] tcp_v6_rcv at ffffffff8166e1b2
 #9 [ffff8808fba03cc0] ip6_input_finish at ffffffff81643712
#10 [ffff8808fba03d00] ip6_input at ffffffff81643fe3
#11 [ffff8808fba03d58] ip6_rcv_finish at ffffffff81643518
#12 [ffff8808fba03d70] ipv6_rcv at ffffffff81643d99
#13 [ffff8808fba03df0] __netif_receive_skb_core at ffffffff81586f22
#14 [ffff8808fba03e60] __netif_receive_skb at ffffffff81587188
#15 [ffff8808fba03e80] process_backlog at ffffffff8158841e
#16 [ffff8808fba03ec0] net_rx_action at ffffffff8158799d
#17 [ffff8808fba03f40] __do_softirq at ffffffff81090b4f
#18 [ffff8808fba03fb0] call_softirq at ffffffff816b6b1c
--- <IRQ stack> ---
#19 [ffff880c1fcb3980] local_bh_enable at ffffffff81090017
#20 [ffff880c1fcb3990] __dev_queue_xmit at ffffffff815895a5
#21 [ffff880c1fcb39e8] local_bh_enable at ffffffff81090017
#22 [ffff880c1fcb39f8] ip6_finish_output2 at ffffffff816407b1
#23 [ffff880c1fcb3a78] ip6_finish_output at ffffffff81642cbc
#24 [ffff880c1fcb3aa0] ip6_output at ffffffff81642d77
#25 [ffff880c1fcb3b00] ip6_xmit at ffffffff81640039
#26 [ffff880c1fcb3ba8] inet6_csk_xmit at ffffffff81673059
#27 [ffff880c1fcb3c48] tcp_transmit_skb at ffffffff815e7c9f
#28 [ffff880c1fcb3cb8] tcp_connect at ffffffff815e97ed
#29 [ffff880c1fcb3d38] tcp_v6_connect at ffffffff8166c106
#30 [ffff880c1fcb3e08] __inet_stream_connect at ffffffff81605725
#31 [ffff880c1fcb3e80] inet_stream_connect at ffffffff816059d8
#32 [ffff880c1fcb3eb0] SYSC_connect at ffffffff8156a497
#33 [ffff880c1fcb3f70] sys_connect at ffffffff8156b29e
#34 [ffff880c1fcb3f80] system_call_fastpath at ffffffff816b5089
 RIP: 00007f9e58f319d0 RSP: 00007ffece4029e0 RFLAGS: 00010246
 RAX: 000000000000002a RBX: ffffffff816b5089 RCX: 00007f9e59144c28
 RDX: 000000000000001c RSI: 00007ffece4030d0 RDI: 0000000000000003
 RBP: 0000000000000000 R8: 0000000000000000 R9: 00000000032dfb60
 R10: 0000000000000006 R11: 0000000000000246 R12: ffffffff8156b29e
 R13: ffff880c1fcb3f78 R14: 0000000002b56510 R15: 00000000025950a0
 ORIG_RAX: 000000000000002a CS: 0033 SS: 002b


https://pastebin.ca/3955185 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-12-06-02:05:41/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Wed Dec 6 02:04:35 2017
 UPTIME: 15 days, 00:23:09
LOAD AVERAGE: 6.23, 4.95, 3.77
 TASKS: 726
 NODENAME: server
 RELEASE: 3.10.0-693.5.2.el7.x86_64
 VERSION: #1 SMP Fri Oct 20 20:32:50 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 00000000000008d0"


bt output



PID: 31570 TASK: ffff8811f9f9dee0 CPU: 11 COMMAND: "mysqld"
 #0 [ffff8800988ff340] machine_kexec at ffffffff8105c4cb
 #1 [ffff8800988ff3a0] __crash_kexec at ffffffff81104a42
 #2 [ffff8800988ff470] crash_kexec at ffffffff81104b30
 #3 [ffff8800988ff488] oops_end at ffffffff816ad338
 #4 [ffff8800988ff4b0] no_context at ffffffff8169d35a
 #5 [ffff8800988ff500] __bad_area_nosemaphore at ffffffff8169d3f0
 #6 [ffff8800988ff548] bad_area at ffffffff8169d714
 #7 [ffff8800988ff570] __do_page_fault at ffffffff816b02fc
 #8 [ffff8800988ff5d0] do_page_fault at ffffffff816b03a5
 #9 [ffff8800988ff600] page_fault at ffffffff816ac5c8
 [exception RIP: xfs_fs_destroy_inode+78]
 RIP: ffffffffc03d6f3e RSP: ffff8800988ff6b0 RFLAGS: 00010202
 RAX: 0000000000000000 RBX: ffff8809781d00f4 RCX: 0000000000020100
 RDX: 000000000000000b RSI: ffff8809781d01d8 RDI: ffff8809781d0000
 RBP: ffff8800988ff6c8 R8: 0000000000000000 R9: 09781d01f00c0000
 R10: f669e55c17347c03 R11: 0000000000000000 R12: ffff8809781d0000
 R13: ffff8809781d0150 R14: ffff8811f97c2108 R15: ffff880cd7cbfe08
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff8800988ff6d0] evict at ffffffff8121d658
#11 [ffff8800988ff710] iget_locked at ffffffff8121d83e
#12 [ffff8800988ff738] prune_icache_sb at ffffffff8121e834
#13 [ffff8800988ff7f0] vmpressure at ffffffff811f7451
#14 [ffff8800988ff878] do_try_to_free_pages at ffffffff811985a2
#15 [ffff8800988ff8f0] try_to_free_pages at ffffffff811987bc
#16 [ffff8800988ff988] __alloc_pages_slowpath at ffffffff8169fbcb
#17 [ffff8800988ffa78] __alloc_pages_nodemask at ffffffff8118cdb5
#18 [ffff8800988ffb28] alloc_pages_current at ffffffff811d1078
#19 [ffff8800988ffb70] alloc_skb_with_frags at ffffffff81573cbd
#20 [ffff8800988ffbc0] sock_alloc_send_pskb at ffffffff8156c9b9
#21 [ffff8800988ffc48] unix_stream_sendmsg at ffffffff8163aef0
#22 [ffff8800988ffcf8] sock_sendmsg at ffffffff8156a580
#23 [ffff8800988ffe58] SYSC_sendto at ffffffff8156a731
#24 [ffff8800988fff70] sys_sendto at ffffffff8156b2ce
#25 [ffff8800988fff80] system_call_fastpath at ffffffff816b5089
 RIP: 00007fca94f07c0b RSP: 00007fca7fed7308 RFLAGS: 00000202
 RAX: 000000000000002c RBX: ffffffff816b5089 RCX: 00000000014c8a7c
 RDX: 0000000000004000 RSI: 00007fca6ebdb008 RDI: 0000000000000163
 RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000000
 R10: 0000000000000040 R11: 0000000000000246 R12: ffffffff8156b2ce
 R13: ffff8800988fff78 R14: 00007fca6ebef238 R15: 0000000000004000
 ORIG_RAX: 000000000000002c CS: 0033 SS: 002b


https://pastebin.ca/3955186 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-14-04:21:30/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Tue Nov 14 04:20:23 2017
 UPTIME: 09:25:26
LOAD AVERAGE: 2.21, 1.47, 1.71
 TASKS: 788
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008"


bt output



PID: 29737 TASK: ffff8811f6346eb0 CPU: 3 COMMAND: "kworker/3:0"
 #0 [ffff8811ac617948] machine_kexec at ffffffff8105c4cb
 #1 [ffff8811ac6179a8] __crash_kexec at ffffffff81104a32
 #2 [ffff8811ac617a78] crash_kexec at ffffffff81104b20
 #3 [ffff8811ac617ab8] no_context at ffffffff8169d2ba
 #4 [ffff8811ac617b08] no_context at ffffffff8169d350
 #5 [ffff8811ac617b50] __bad_area_nosemaphore at ffffffff8169d4ba
 #6 [ffff8811ac617b60] __do_page_fault at ffffffff816b017e
 #7 [ffff8811ac617bc0] __do_page_fault at ffffffff816b0325
 #8 [ffff8811ac617bf0] general_protection at ffffffff816ac548
 [exception RIP: xlog_write+772]
 RIP: ffffffffc03f8644 RSP: ffff8811ac617ca0 RFLAGS: 00010286
 RAX: 0000000000000000 RBX: ffff8811af615870 RCX: 0000000000000000
 RDX: 00000000000261a0 RSI: 0000000000000000 RDI: 0000000000005e6c
 RBP: ffff8811ac617d38 R8: 0000000000000000 R9: ffffc9000d576078
 R10: ffffc9000d57606c R11: ffff8808f53ad800 R12: 0000000000007e00
 R13: 00000000000000d0 R14: 0000000000000000 R15: ffff880c9c7bb000
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff8811ac617d40] xlog_cil_push at ffffffffc03fa1f8 [xfs]
#10 [ffff8811ac617e10] xlog_cil_push_work at ffffffffc03fa395 [xfs]
#11 [ffff8811ac617e20] process_one_work at ffffffff810a881a
#12 [ffff8811ac617e68] worker_thread at ffffffff810a94e6
#13 [ffff8811ac617ec8] kthread at ffffffff810b098f
#14 [ffff8811ac617f50] save_rest at ffffffff816b4f58


https://pastebin.ca/3955187 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-13-18:50:12/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Mon Nov 13 18:49:05 2017
 UPTIME: 10 days, 17:30:03
LOAD AVERAGE: 2.10, 2.43, 2.33
 TASKS: 785
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000020"


btoutput



PID: 27951 TASK: ffff8808f1b25ee0 CPU: 16 COMMAND: "php-cgi"
 #0 [ffff880113643a90] machine_kexec at ffffffff8105c4cb
 #1 [ffff880113643af0] __crash_kexec at ffffffff81104a32
 #2 [ffff880113643bc0] crash_kexec at ffffffff81104b20
 #3 [ffff880113643c00] no_context at ffffffff8169d2ba
 #4 [ffff880113643c50] no_context at ffffffff8169d350
 #5 [ffff880113643c98] mm_fault_error at ffffffff8169d674
 #6 [ffff880113643cc0] __do_page_fault at ffffffff816b027c
 #7 [ffff880113643d20] __do_page_fault at ffffffff816b0325
 #8 [ffff880113643d50] general_protection at ffffffff816ac548
 [exception RIP: xfs_free_eofblocks+91]
 RIP: ffffffffc03cf8db RSP: ffff880113643e00 RFLAGS: 00010212
 RAX: 0000000000000000 RBX: ffff8811f3bdcec0 RCX: 000000000000000c
 RDX: 0000000000000fff RSI: 0000000000000001 RDI: ffff8811f3bdcec0
 RBP: ffff880113643e58 R8: 0000000000000000 R9: 0000000000000000
 R10: ffff8811f3bdd010 R11: ffff880017c7b710 R12: 0000000000015a0b
 R13: ffff8808f7346000 R14: ffff8808f4ba00c0 R15: ffff880170ad2d20
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #9 [ffff880113643e60] xfs_release at ffffffffc03e7105 [xfs]
#10 [ffff880113643e88] xfs_file_release at ffffffffc03d65a5 [xfs]
#11 [ffff880113643e98] delayed_fput at ffffffff81202fc9
#12 [ffff880113643ee0] alloc_file at ffffffff8120322e
#13 [ffff880113643ef0] task_work_run at ffffffff810ad247
#14 [ffff880113643f30] do_notify_resume at ffffffff8102ab62
#15 [ffff880113643f50] int_with_check at ffffffff816b52bd
 RIP: 00007fd2f72abe90 RSP: 00007ffe5d25d848 RFLAGS: 00000246
 RAX: 0000000000000000 RBX: 000055bac5cda3c0 RCX: ffffffffffffffff
 RDX: 0000000000000008 RSI: 0000000000000001 RDI: 0000000000000003
 RBP: 0000000000000000 R8: 000055bac5cda4a0 R9: 00007fd2fa6bb840
 R10: 000000000000000d R11: 0000000000000246 R12: 0000000000000000
 R13: 0000000000000001 R14: 00007fd2e8771f58 R15: 0000000000000000
 ORIG_RAX: 0000000000000003 CS: 0033 SS: 002b


https://pastebin.ca/3955188 



 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-11-03-01:14:25/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Fri Nov 3 01:13:19 2017
 UPTIME: 3 days, 18:53:59
LOAD AVERAGE: 2.66, 2.58, 2.16
 TASKS: 744
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "BUG: unable to handle kernel NULL pointer dereference at (null)"


bt output



PID: 144 TASK: ffff8808faf52f70 CPU: 7 COMMAND: "kswapd1"
 #0 [ffff8808f9e0f7e8] machine_kexec at ffffffff8105c4cb
 #1 [ffff8808f9e0f848] __crash_kexec at ffffffff81104a32
 #2 [ffff8808f9e0f918] crash_kexec at ffffffff81104b20
 #3 [ffff8808f9e0f958] no_context at ffffffff8169d2ba
 #4 [ffff8808f9e0f9a8] no_context at ffffffff8169d350
 #5 [ffff8808f9e0f9f0] __bad_area_nosemaphore at ffffffff8169d4ba
 #6 [ffff8808f9e0fa00] __do_page_fault at ffffffff816b017e
 #7 [ffff8808f9e0fa08] __radix_tree_create at ffffffff81328a9e
 #8 [ffff8808f9e0fa60] __do_page_fault at ffffffff816b0325
 #9 [ffff8808f9e0fa90] general_protection at ffffffff816ac548
 [exception RIP: crc32_generic_combine+89]
 RIP: ffffffff8133db39 RSP: ffff8808f9e0fb40 RFLAGS: 00010207
 RAX: 0000000000000000 RBX: ffff88010b138008 RCX: dead000000000200
 RDX: 0000000000000000 RSI: ffff88010b8bc570 RDI: ffff88010b138008
 RBP: ffff8808f9e0fb40 R8: e018000000000000 R9: 010b8bc5700c0000
 R10: fed6747d72e55c03 R11: 0000000000000000 R12: ffff88010b138000
 R13: ffff8808f47dbcb8 R14: ffff8811f973f108 R15: ffff880143a76b48
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#10 [ffff8808f9e0fb48] file_has_perm at ffffffff812b752d
#11 [ffff8808f9e0fb88] evict at ffffffff8121d5e2
#12 [ffff8808f9e0fbb8] iget_locked at ffffffff8121d81a
#13 [ffff8808f9e0fbe0] iget_locked at ffffffff8121d8ce
#14 [ffff8808f9e0fc08] new_inode_pseudo at ffffffff8121e8c4
#15 [ffff8808f9e0fc70] ns_set_super at ffffffff81203878
#16 [ffff8808f9e0fca8] shrink_slab at ffffffff81195413
#17 [ffff8808f9e0fcc0] vmpressure_register_event at ffffffff811f7547
#18 [ffff8808f9e0fd48] balance_pgdat at ffffffff81199081
#19 [ffff8808f9e0fe20] kswapd at ffffffff81199323
#20 [ffff8808f9e0fe78] wake_up_atomic_t at ffffffff810b1910
#21 [ffff8808f9e0fea8] balance_pgdat at ffffffff811991b0
#22 [ffff8808f9e0fec8] kthread at ffffffff810b098f
#23 [ffff8808f9e0ff50] save_rest at ffffffff816b4f58

https://pastebin.ca/3955189

 KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.11.1.el7.x86_64/vmlinux
 DUMPFILE: /var/crash/127.0.0.1-2017-10-30-07:14:38/vmcore [PARTIAL DUMP]
 CPUS: 24
 DATE: Mon Oct 30 06:13:31 2017
 UPTIME: 3 days, 06:21:18
LOAD AVERAGE: 1.03, 1.19, 1.36
 TASKS: 707
 NODENAME: server
 RELEASE: 3.10.0-693.2.2.el7.x86_64
 VERSION: #1 SMP Tue Sep 12 22:26:13 UTC 2017
 MACHINE: x86_64 (2666 Mhz)
 MEMORY: 72 GB
 PANIC: "general protection fault: 0000 [#1] SMP "


bt output



PID: 32005 TASK: ffff8808f8bfeeb0 CPU: 4 COMMAND: "vtund"
 #0 [ffff8804623fb950] machine_kexec at ffffffff8105c4cb
 #1 [ffff8804623fb9b0] __crash_kexec at ffffffff81104a32
 #2 [ffff8804623fba80] crash_kexec at ffffffff81104b20
 #3 [ffff8804623fbac0] die at ffffffff8102e97b
 #4 [ffff8804623fbaf0] do_general_protection at ffffffff816acc3e
 #5 [ffff8804623fbb20] xen_int3 at ffffffff816ac4e8
 [exception RIP: memcmp]
 RIP: ffffffff8132b980 RSP: ffff8804623fbbd0 RFLAGS: 00010286
 RAX: ffff8801f3cd7048 RBX: 000000000000000a RCX: ffff8806917dc048
 RDX: ffff8808fb333800 RSI: ffff8808f9f482e8 RDI: 0236303633043a00
 RBP: ffff8804623fbc40 R8: 0000000000000036 R9: 0000000000000000
 R10: ffff88017fc03400 R11: 0000000000000000 R12: ffffffff8197ea30
 R13: 0236303633043a00 R14: ffff8808fb333868 R15: ffff8808a4f6a300
 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
 #6 [ffff8804623fbbd0] insert_header at ffffffff8127cb99
 #7 [ffff8804623fbc48] register_leaf_sysctl_tables at ffffffff8127d28b
 #8 [ffff8804623fbc60] bstr_printf at ffffffff8132fb59
 #9 [ffff8804623fbd40] ipv6_add_dev at ffffffff8164784a
#10 [ffff8804623fbd70] addrconf_notify at ffffffff8164c9c9
#11 [ffff8804623fbd78] sysfs_slab_alias at ffffffff811df8d6
#12 [ffff8804623fbd88] dropmon_net_event at ffffffff815ab686
#13 [ffff8804623fbde0] trace_do_page_fault at ffffffff816b051c
#14 [ffff8804623fbe18] raw_notifier_call_chain at ffffffff810b68d6
#15 [ffff8804623fbe28] call_netdevice_notifiers_info at ffffffff8158306d
#16 [ffff8804623fbe50] register_netdevice at ffffffff8158c576
#17 [ffff8804623fbe88] __tun_chr_ioctl at ffffffffc07fca8e [tun]
#18 [ffff8804623fbf20] tun_chr_compat_ioctl at ffffffffc07fd10b [tun]
#19 [ffff8804623fbf30] compat_sys_ioctl at ffffffff8125d73b
#20 [ffff8804623fbf80] cstar_tracesys at ffffffff816b746c
 RIP: 00000000080d458c RSP: 00000000fff1980c RFLAGS: 00000246
 RAX: ffffffffffffffda RBX: ffffffff816b746c RCX: 00000000400454ca
 RDX: 00000000fff19840 RSI: 00000000fff19840 RDI: 00000000fff19860
 RBP: 00000000fff19878 R8: 0000000000000000 R9: 0000000000000000
 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 ORIG_RAX: 0000000000000036 CS: 0023 SS: 002b




centos kernel-panic





share|improve this question













share|improve this question




share|improve this question








edited Jan 4 at 19:44

























asked Jan 4 at 1:56









Jouni Karlsson

63




63











  • I suggest you directly put all the relevant information here. With proper formatting you can show all of them in a ordered manner without worrying that it would distract people from your main question.
    – Weijun Zhou
    Jan 4 at 3:25
















  • I suggest you directly put all the relevant information here. With proper formatting you can show all of them in a ordered manner without worrying that it would distract people from your main question.
    – Weijun Zhou
    Jan 4 at 3:25















I suggest you directly put all the relevant information here. With proper formatting you can show all of them in a ordered manner without worrying that it would distract people from your main question.
– Weijun Zhou
Jan 4 at 3:25




I suggest you directly put all the relevant information here. With proper formatting you can show all of them in a ordered manner without worrying that it would distract people from your main question.
– Weijun Zhou
Jan 4 at 3:25










1 Answer
1






active

oldest

votes

















up vote
1
down vote













Do a memtest. Your issue is memory corruption. Memory errors can cause random, hard to diagnose errors.






share|improve this answer






















  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review
    – jasonwryan
    Jan 4 at 3:05










  • Thanks for the feedback. I've edited my answer to be less of a commentary.
    – Free Bullets
    Jan 4 at 6:27










  • Server is running ECC memory and should by my understanding exhibit Machine-check exceptions at least some times before failing. No indication of these in logs. Thanks.
    – Jouni Karlsson
    Jan 4 at 19:41










Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);








 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f414681%2fdetermining-cause-of-kernel-panics%23new-answer', 'question_page');

);

Post as a guest

















1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
1
down vote













Do a memtest. Your issue is memory corruption. Memory errors can cause random, hard to diagnose errors.






share|improve this answer






















  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review
    – jasonwryan
    Jan 4 at 3:05










  • Thanks for the feedback. I've edited my answer to be less of a commentary.
    – Free Bullets
    Jan 4 at 6:27










  • Server is running ECC memory and should by my understanding exhibit Machine-check exceptions at least some times before failing. No indication of these in logs. Thanks.
    – Jouni Karlsson
    Jan 4 at 19:41














up vote
1
down vote













Do a memtest. Your issue is memory corruption. Memory errors can cause random, hard to diagnose errors.






share|improve this answer






















  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review
    – jasonwryan
    Jan 4 at 3:05










  • Thanks for the feedback. I've edited my answer to be less of a commentary.
    – Free Bullets
    Jan 4 at 6:27










  • Server is running ECC memory and should by my understanding exhibit Machine-check exceptions at least some times before failing. No indication of these in logs. Thanks.
    – Jouni Karlsson
    Jan 4 at 19:41












up vote
1
down vote










up vote
1
down vote









Do a memtest. Your issue is memory corruption. Memory errors can cause random, hard to diagnose errors.






share|improve this answer














Do a memtest. Your issue is memory corruption. Memory errors can cause random, hard to diagnose errors.







share|improve this answer














share|improve this answer



share|improve this answer








edited Jan 4 at 6:26

























answered Jan 4 at 2:33









Free Bullets

37526




37526











  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review
    – jasonwryan
    Jan 4 at 3:05










  • Thanks for the feedback. I've edited my answer to be less of a commentary.
    – Free Bullets
    Jan 4 at 6:27










  • Server is running ECC memory and should by my understanding exhibit Machine-check exceptions at least some times before failing. No indication of these in logs. Thanks.
    – Jouni Karlsson
    Jan 4 at 19:41
















  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review
    – jasonwryan
    Jan 4 at 3:05










  • Thanks for the feedback. I've edited my answer to be less of a commentary.
    – Free Bullets
    Jan 4 at 6:27










  • Server is running ECC memory and should by my understanding exhibit Machine-check exceptions at least some times before failing. No indication of these in logs. Thanks.
    – Jouni Karlsson
    Jan 4 at 19:41















This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review
– jasonwryan
Jan 4 at 3:05




This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review
– jasonwryan
Jan 4 at 3:05












Thanks for the feedback. I've edited my answer to be less of a commentary.
– Free Bullets
Jan 4 at 6:27




Thanks for the feedback. I've edited my answer to be less of a commentary.
– Free Bullets
Jan 4 at 6:27












Server is running ECC memory and should by my understanding exhibit Machine-check exceptions at least some times before failing. No indication of these in logs. Thanks.
– Jouni Karlsson
Jan 4 at 19:41




Server is running ECC memory and should by my understanding exhibit Machine-check exceptions at least some times before failing. No indication of these in logs. Thanks.
– Jouni Karlsson
Jan 4 at 19:41












 

draft saved


draft discarded


























 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f414681%2fdetermining-cause-of-kernel-panics%23new-answer', 'question_page');

);

Post as a guest
































































-

Popular posts from this blog

How to check contact read email or not when send email to Individual?

Image
Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this questi...
Read more

Displaying single band from multi-band raster using QGIS

Image
Clash Royale CLAN TAG #URR8PPP 1 How can I extract a single band from multi-band raster in QGIS? I have an remote sensed image which has 6 bands (including NDVI band), I want to display each band separately, but have no idea how to do. I have seen some questions similar here but none worked for me. The original image (has 6 bands) is: I want to display the band 6 which should be like this: But I tried gdal_translate, and couldn't get the correct result. What I have got is: qgis raster multi-band share | improve this question edited Mar 5 at 0:53 Summer asked Mar 4 at 6:42 Summer Summer 23 6 Is this any help gis.stackexchange.com/questions/220658/… ? if not gis.stackexchange.com/questions/62133/… might help. – Michael Stimson Mar 4 at 6:46 Thanks for answering but when I used gdal_translate, qgis showed that 'Error 4: Kayena.tif: No such file or directory". Would you know how to fi...
Read more

How many registers does an x86_64 CPU actually have?

Image
Clash Royale CLAN TAG #URR8PPP up vote 2 down vote favorite I am currently learning reverse engineering and am studying the flags register. I had in my mind that rflags was just another name for one of the 16 general purpose registers, for example rax or rbx . But it looks like rflags is actually an additional register. So that makes 17 registers in total... how many more could there be? I have spent at least an hour on this and found numerous different answers. The best answer so far is this, which says that there are 40 registers in total. 16 General Purpose Registers 2 Status Registers 6 Code Segment Registers 16 SSE Registers 8 FPU/MMX Registers But if I add that up, I get 48. Could anybody provide an official answer on how many registers an x86_64 CPU has (e.g. an Intel i7). Additionally, I have seen references to 'hardware' and 'architectural' registers. What are those registers and how many are there? register x86-64 share | improve this...
Read more