Kernel Crash in SCTP Code - Red Hat Solutions Guide

https://access.redhat.com/solutions/6016061

Red Hat Insights can detect this issue

Proactively detect and remediate issues impacting your systems.

[View matching systems and remediation](http://console.redhat.com/insights/advisor/recommendations/kernel_crash_with_sctp KERNEL_CRASH_IN_SCTP_FOR_EDGE_DEFAULT_KERNEL)

Environment

  • Red Hat Enterprise Linux (RHEL) 8.3
  • Red Hat Enterprise Linux (RHEL) 8.4
  • Red Hat Enterprise Linux (RHEL) 7.9 and earlier
  • SCTP (Streaming Control Transmission Protocol)

Issue

  • RHEL 8.3/RHEL 7.9 kernel crashes in sctp code with one of these RIPs:
[exception RIP: sctp_ulpevent_notify_peer_addr_change+0x30]
    RIP: ffffffffc06b76c0  RSP: ffff9d2133b83b08  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: 0000000000001000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000001  RDI: ffff9d20f24a4400
    RBP: ffff9d21245d8000   R8: 0000000000000003   R9: ffff9d2133b83ce0
    R10: ffffffffc06d8d10  R11: ffff9d2133b83e20  R12: 0000000000000000
    R13: ffff9d21245d8000  R14: 0000000000000003  R15: ffff9d2133b83ce0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff9d2133b83bc0] sctp_assoc_control_transport at ffffffffc06aef4b [sctp]
 #8 [ffff9d2133b83c00] sctp_do_8_2_transport_strike at ffffffffc06aa712 [sctp]
 #9 [ffff9d2133b83c20] sctp_cmd_interpreter at ffffffffc06abbdf [sctp]
#10 [ffff9d2133b83ca0] sctp_do_sm at ffffffffc06aa873 [sctp]
#11 [ffff9d2133b83e68] sctp_generate_timeout_event at ffffffffc06aaec1 [sctp]
#12 [ffff9d2133b83ea8] call_timer_fn at ffffffffb353dfbd
#13 [ffff9d2133b83ed8] run_timer_softirq at ffffffffb353eb18
#14 [ffff9d2133b83f68] __softirqentry_text_start at ffffffffb40000e4
#15 [ffff9d2133b83fc8] irq_exit at ffffffffb34bc217
#16 [ffff9d2133b83fd8] smp_apic_timer_interrupt at ffffffffb3e027e4
#17 [ffff9d2133b83ff0] apic_timer_interrupt at ffffffffb3e01d6f
 [exception RIP: sctp_generate_heartbeat_event+0x24]
    RIP: ffffffffc0ab2b54  RSP: ffff9384afc03e60  RFLAGS: 00010286
    RAX: dead000000000200  RBX: 0000000000000100  RCX: 0000000000000240
    RDX: 0000000100133380  RSI: ffffffffc0ab2b30  RDI: ffff9383dcd415c8
    RBP: ffff9383dcd415c8   R8: ffff9384afc1af60   R9: ffff9384afc03ef0
    R10: ffff9384afc1aaf0  R11: 003b9aca00000000  R12: ffff9383dcd415c8
    R13: ffff9383dcd41400  R14: ffff9383dcd415c8  R15: 0040f03740004845
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #5 [ffff9384afc03ea8] call_timer_fn at ffffffffb333dfbd
 #6 [ffff9384afc03ed8] run_timer_softirq at ffffffffb333eb18
 #7 [ffff9384afc03f68] __softirqentry_text_start at ffffffffb3e000e4
 #8 [ffff9384afc03fc8] irq_exit at ffffffffb32bc217
 #9 [ffff9384afc03fd8] smp_apic_timer_interrupt at ffffffffb3c027e4
#10 [ffff9384afc03ff0] apic_timer_interrupt at ffffffffb3c01d6f
  [exception RIP: unknown or invalid address]
    RIP: 0000000000000000  RSP: ffff89be6fd83ea8  RFLAGS: 00010206
    RAX: dead000000000200  RBX: 0000000000000100  RCX: 0000000000000240
    RDX: 0000000101abc440  RSI: 0000000000000000  RDI: ffff89be3dccf5c8
    RBP: 0000000000000000   R8: ffff89be6fd9af78   R9: ffff89be6fd83ef0
    R10: ffff89be6fd9aaf0  R11: 0000000000000000  R12: ffff89be3dccf5c8
    R13: 0000000000000000  R14: ffff89be3dccf5c8  R15: ffffffffb0805100
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff89be6fd83ea8] call_timer_fn at ffffffffaf73dfbd
 #8 [ffff89be6fd83ed8] run_timer_softirq at ffffffffaf73eb18
 #9 [ffff89be6fd83f68] __softirqentry_text_start at ffffffffb02000e4
#10 [ffff89be6fd83fc8] irq_exit at ffffffffaf6bc217
#11 [ffff89be6fd83fd8] smp_apic_timer_interrupt at ffffffffb00027e4
#12 [ffff89be6fd83ff0] apic_timer_interrupt at ffffffffb0001d6f
[460769.460712] general protection fault: 0000 [#1] SMP
[460769.464454] CPU: 16 PID: 4329 Comm: udrad Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
[460769.465018] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 05/17/2022
[460769.465575] task: ffff9ed595e21080 ti: ffff9ed462e58000 task.ti: ffff9ed462e58000
[460769.466136] RIP: 0010:[<ffffffffc03854bf>]  [<ffffffffc03854bf>] sctp_assoc_control_transport+0x20f/0x2a0 [sctp]
[460769.466756] RSP: 0018:ffff9ed59fa03ac0  EFLAGS: 00010286
[460769.467380] RAX: 3130636e6d2e736d RBX: 0000000000000000 RCX: 0000000000000000
[460769.468014] RDX: 0000000000000001 RSI: ffff9ecdde6c8890 RDI: ffff9ed59fa03ac0
[460769.468652] RBP: ffff9ed59fa03b70 R08: 0000000000000003 R09: ffff9ed59fa03c90
[460769.469261] R10: ffff9ed59fa03ac0 R11: 0000000000000005 R12: 0000000000000001
[460769.469885] R13: 0000000000000000 R14: 0000000000000003 R15: ffff9ed455f4f000
[460769.470494] FS:  00007fa7c7ff7700(0000) GS:ffff9ed59fa00000(0000) knlGS:0000000000000000
[460769.471122] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[460769.471783] CR2: 00007fa68e125131 CR3: 000000016efea000 CR4: 00000000007607e0
[460769.472371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[460769.472959] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[460769.473544] PKRU: 55555554
[460769.474124] Call Trace:
[460769.474727]  <IRQ>
[460769.474740]  [<ffffffffc0380a26>] sctp_do_8_2_transport_strike.isra.18+0x106/0x290 [sctp]
[460769.475927]  [<ffffffffc0382385>] sctp_cmd_interpreter.isra.22+0xf55/0x1450 [sctp]
[460769.476526]  [<ffffffffc0380c91>] sctp_do_sm+0xe1/0x350 [sctp]
[460769.477160]  [<ffffffffc03a2280>] ? sctp_oname+0x30/0x30 [sctp]
[460769.477828]  [<ffffffffc0381305>] sctp_generate_timeout_event+0xc5/0x110 [sctp]
[460769.478505]  [<ffffffffc03813d0>] ? sctp_generate_t4_rto_event+0x20/0x20 [sctp]
[460769.479208]  [<ffffffffc03813e3>] sctp_generate_t2_shutdown_event+0x13/0x20 [sctp]
[460769.479863]  [<ffffffff8f6abd58>] call_timer_fn+0x38/0x110
[460769.480543]  [<ffffffffc03813d0>] ? sctp_generate_t4_rto_event+0x20/0x20 [sctp]
[460769.481231]  [<ffffffff8f6ae1ed>] run_timer_softirq+0x24d/0x300
[460769.481866]  [<ffffffff8f6a4b95>] __do_softirq+0xf5/0x280
[460769.482511]  [<ffffffff8fd974ec>] call_softirq+0x1c/0x30
[460769.483178]  [<ffffffff8f62f715>] do_softirq+0x65/0xa0
[460769.483836]  [<ffffffff8f6a4f15>] irq_exit+0x105/0x110
[460769.484499]  [<ffffffff8fd98a88>] smp_apic_timer_interrupt+0x48/0x60
[460769.485158]  [<ffffffff8fd94fba>] apic_timer_interrupt+0x16a/0x170
[460769.498412] Code: 00 00 00 41 bc 01 00 00 00 4c 8d 95 50 ff ff ff 31 c0 b9 10 00 00 00 48 81 c6 90 00 00 00 4c 89 d7 f3 48 ab 48 8b 46 20 4c 89 d7 <48> 63 90 bc 00 00 00 e8 95 19 61 cf 31 d2 41 b9 20 00 00 00 41
[460769.499550] RIP  [<ffffffffc03854bf>] sctp_assoc_control_transport+0x20f/0x2a0 [sctp]
[460769.500150]  RSP <ffff9ed59fa03ac0>
[702302.031564] general protection fault: 0000 [#1] SMP
[702302.031839] CPU: 14 PID: 0 Comm: swapper/14 Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
[702302.031864] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 05/17/2022
[702302.031888] task: ffff97bab2f51080 ti: ffff97bab2f5c000 task.ti: ffff97bab2f5c000
[702302.031915] RIP: 0010:[<ffffffffc038f991>]  [<ffffffffc038f991>] sctp_do_8_2_transport_strike.isra.18+0x71/0x290 [sctp]
[702302.031952] RSP: 0018:ffff97c15f983b80  EFLAGS: 00010246
[702302.031968] RAX: 6d2e303130636e6d RBX: ffff97b96590fc00 RCX: 0000000000000001
[702302.031998] RDX: ce665c60626ac6c6 RSI: ffff97b96590fc00 RDI: ffff97baa4bb5000
[702302.032030] RBP: ffff97c15f983b98 R08: 0000000000000003 R09: ffff97c15f983c90
[702302.032050] R10: ffff97babfc03600 R11: 0000000000000005 R12: ffff97baa4bb5000
[702302.032071] R13: 0000000000000000 R14: 0000000000000003 R15: ffff97c15f983c90
[702302.032103] FS:  0000000000000000(0000) GS:ffff97c15f980000(0000) knlGS:0000000000000000
[702302.032125] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[702302.032141] CR2: 00007f066ee55ca0 CR3: 0000000eae410000 CR4: 00000000007607e0
[702302.032161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[702302.032181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[702302.032201] PKRU: 00000000
[702302.032210] Call Trace:
[702302.032219]  <IRQ>
[702302.032233]  [<ffffffffc0391385>] sctp_cmd_interpreter.isra.22+0xf55/0x1450 [sctp]
[702302.032258]  [<ffffffffc038fc91>] sctp_do_sm+0xe1/0x350 [sctp]
[702302.032281]  [<ffffffffc03b1280>] ? sctp_oname+0x30/0x30 [sctp]
[702302.032301]  [<ffffffffc0390305>] sctp_generate_timeout_event+0xc5/0x110 [sctp]
[702302.032324]  [<ffffffffc03903d0>] ? sctp_generate_t4_rto_event+0x20/0x20 [sctp]
[702302.032347]  [<ffffffffc03903e3>] sctp_generate_t2_shutdown_event+0x13/0x20 [sctp]
[702302.032371]  [<ffffffff85aabd58>] call_timer_fn+0x38/0x110
[702302.032389]  [<ffffffffc03903d0>] ? sctp_generate_t4_rto_event+0x20/0x20 [sctp]
[702302.032411]  [<ffffffff85aae1ed>] run_timer_softirq+0x24d/0x300
[702302.032430]  [<ffffffff85aa4b95>] __do_softirq+0xf5/0x280
[702302.032448]  [<ffffffff861974ec>] call_softirq+0x1c/0x30
[702302.032466]  [<ffffffff85a2f715>] do_softirq+0x65/0xa0
[702302.032492]  [<ffffffff85aa4f15>] irq_exit+0x105/0x110
[702302.032518]  [<ffffffff86198a88>] smp_apic_timer_interrupt+0x48/0x60
[702302.032539]  [<ffffffff86194fba>] apic_timer_interrupt+0x16a/0x170
[702302.040088] Code: 09 83 f8 02 0f 84 70 01 00 00 85 c0 75 7c 45 85 ed 74 06 f6 43 24 02 74 35 48 8b 83 c0 00 00 00 48 8d 14 00 48 8b 83 b8 00 00 00 <48> 8b 80 68 02 00 00 48 39 c2 48 0f 46 c2 48 89 83 c0 00 00 00
[702302.041559] RIP  [<ffffffffc038f991>] sctp_do_8_2_transport_strike.isra.18+0x71/0x290 [sctp]
[702302.042265]  RSP <ffff97c15f983b80>

Resolution

Upgrade to one of the following packages or later:

Red Hat Enterprise Linux release Package Errata
8.5 kernel-4.18.0-348.el8 RHSA-2021:4356
8.4 kernel-4.18.0-305.7.1.el8_4 RHSA-2021:2570
7.9 kernel-3.10.0-1160.88.1.el7 RHSA-2023:1091

Workaround - Earlier package

RHEL 8.2 (kernel-4.18.0-193.el8) or previous kernels are not affected by this bug.

Root Cause

It was found out that all the crashes have the same problem which is use after free of sctp_transport structure. The problem seems to happen when an existing SCTP socket in shutdown state gets a new INIT/COOKIE-ECHO packet (tuple reuse) and proper validation of SCTP state is not performed and leading to crash.

The below commit is identified to fix this issue:

Diagnostic Steps

  • Analysis:
[460769.460712] general protection fault: 0000 [#1] SMP
[460769.461260] Modules linked in: sctp bonding dm_mirror dm_region_hash dm_log dm_mod vfat fat ipmi_ssif skx_edac nfit libnvdimm intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure sg mei_me lpc_ich mei hpwdt hpilo wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm uas drm crct10dif_pclmul igb crct10dif_common crc32c_intel smartpqi usb_storage scsi_transport_sas ptp pps_core dca i2c_algo_bit drm_panel_orientation_quirks
[460769.464454] CPU: 16 PID: 4329 Comm: udrad Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
[460769.465018] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 05/17/2022
[460769.465575] task: ffff9ed595e21080 ti: ffff9ed462e58000 task.ti: ffff9ed462e58000
[460769.466136] RIP: 0010:[<ffffffffc03854bf>]  [<ffffffffc03854bf>] sctp_assoc_control_transport+0x20f/0x2a0 [sctp]
[460769.466756] RSP: 0018:ffff9ed59fa03ac0  EFLAGS: 00010286
[460769.467380] RAX: 3130636e6d2e736d RBX: 0000000000000000 RCX: 0000000000000000
[460769.468014] RDX: 0000000000000001 RSI: ffff9ecdde6c8890 RDI: ffff9ed59fa03ac0
[460769.468652] RBP: ffff9ed59fa03b70 R08: 0000000000000003 R09: ffff9ed59fa03c90
[460769.469261] R10: ffff9ed59fa03ac0 R11: 0000000000000005 R12: 0000000000000001
[460769.469885] R13: 0000000000000000 R14: 0000000000000003 R15: ffff9ed455f4f000
[460769.470494] FS:  00007fa7c7ff7700(0000) GS:ffff9ed59fa00000(0000) knlGS:0000000000000000
[460769.471122] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[460769.471783] CR2: 00007fa68e125131 CR3: 000000016efea000 CR4: 00000000007607e0
[460769.472371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[460769.472959] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[460769.473544] PKRU: 55555554
[460769.474124] Call Trace:
[460769.474727]  <IRQ>
[460769.474740]  [<ffffffffc0380a26>] sctp_do_8_2_transport_strike.isra.18+0x106/0x290 [sctp]
[460769.475927]  [<ffffffffc0382385>] sctp_cmd_interpreter.isra.22+0xf55/0x1450 [sctp]
[460769.476526]  [<ffffffffc0380c91>] sctp_do_sm+0xe1/0x350 [sctp]
[460769.477160]  [<ffffffffc03a2280>] ? sctp_oname+0x30/0x30 [sctp]
[460769.477828]  [<ffffffffc0381305>] sctp_generate_timeout_event+0xc5/0x110 [sctp]
[460769.478505]  [<ffffffffc03813d0>] ? sctp_generate_t4_rto_event+0x20/0x20 [sctp]
[460769.479208]  [<ffffffffc03813e3>] sctp_generate_t2_shutdown_event+0x13/0x20 [sctp]
[460769.479863]  [<ffffffff8f6abd58>] call_timer_fn+0x38/0x110
[460769.480543]  [<ffffffffc03813d0>] ? sctp_generate_t4_rto_event+0x20/0x20 [sctp]
[460769.481231]  [<ffffffff8f6ae1ed>] run_timer_softirq+0x24d/0x300
[460769.481866]  [<ffffffff8f6a4b95>] __do_softirq+0xf5/0x280
[460769.482511]  [<ffffffff8fd974ec>] call_softirq+0x1c/0x30
[460769.483178]  [<ffffffff8f62f715>] do_softirq+0x65/0xa0
[460769.483836]  [<ffffffff8f6a4f15>] irq_exit+0x105/0x110
[460769.484499]  [<ffffffff8fd98a88>] smp_apic_timer_interrupt+0x48/0x60
[460769.485158]  [<ffffffff8fd94fba>] apic_timer_interrupt+0x16a/0x170
[460769.485864]  <EOI>
[460769.485901]  [<ffffffffc02b3d8b>] ? xfs_log_ticket_put+0x2b/0x30 [xfs]
[460769.487238]  [<ffffffff8f826466>] ? kmem_cache_free+0x6/0x200
[460769.487959]  [<ffffffffc02ae0d2>] ? xfs_trans_free_item_desc+0x32/0x40 [xfs]
[460769.488656]  [<ffffffffc02aed6d>] xfs_trans_free_items+0x8d/0xb0 [xfs]
[460769.489311]  [<ffffffffc02b53e8>] xfs_log_commit_cil+0x448/0x660 [xfs]
[460769.490011]  [<ffffffffc02aeebd>] __xfs_trans_commit+0x12d/0x260 [xfs]
[460769.490713]  [<ffffffffc02af2b0>] xfs_trans_commit+0x10/0x20 [xfs]
[460769.491442]  [<ffffffffc029f5b6>] xfs_vn_update_time+0x146/0x150 [xfs]
[460769.492067]  [<ffffffff8f86b888>] update_time+0x28/0xd0
[460769.492674]  [<ffffffff8f86b9d0>] file_update_time+0xa0/0xf0
[460769.493247]  [<ffffffffc0294edf>] xfs_file_aio_write_checks+0x16f/0x1c0 [xfs]
[460769.493810]  [<ffffffffc029575a>] xfs_file_buffered_aio_write+0xca/0x2c0 [xfs]
[460769.494370]  [<ffffffffc0295add>] xfs_file_aio_write+0x18d/0x1b0 [xfs]
[460769.494906]  [<ffffffff8f84d12b>] do_sync_readv_writev+0x7b/0xd0
[460769.495418]  [<ffffffff8f84ed2e>] do_readv_writev+0xce/0x260
[460769.495955]  [<ffffffffc0295950>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
[460769.496452]  [<ffffffff8f84cfd0>] ? do_sync_read+0xe0/0xe0
[460769.496949]  [<ffffffff8f84ef55>] vfs_writev+0x35/0x60
[460769.497438]  [<ffffffff8f84f10f>] SyS_writev+0x7f/0x110
[460769.497927]  [<ffffffff8fd93f92>] system_call_fastpath+0x25/0x2a
[460769.498412] Code: 00 00 00 41 bc 01 00 00 00 4c 8d 95 50 ff ff ff 31 c0 b9 10 00 00 00 48 81 c6 90 00 00 00 4c 89 d7 f3 48 ab 48 8b 46 20 4c 89 d7 <48> 63 90 bc 00 00 00 e8 95 19 61 cf 31 d2 41 b9 20 00 00 00 41
[460769.499550] RIP  [<ffffffffc03854bf>] sctp_assoc_control_transport+0x20f/0x2a0 [sctp]
[460769.500150]  RSP <ffff9ed59fa03ac0>

crash> dis -rl sctp_assoc_control_transport+0x20f|tail
/usr/src/debug/kernel-3.10.0-1160.el7/linux-3.10.0-1160.el7.x86_64/net/sctp/associola.c: 857
0xffffffffc08474fb <sctp_assoc_control_transport+507>:  add    $0x90,%rsi
/usr/src/debug/kernel-3.10.0-1160.el7/linux-3.10.0-1160.el7.x86_64/net/sctp/associola.c: 856
0xffffffffc0847502 <sctp_assoc_control_transport+514>:  mov    %r10,%rdi
0xffffffffc0847505 <sctp_assoc_control_transport+517>:  rep stos %rax,%es:(%rdi)
/usr/src/debug/kernel-3.10.0-1160.el7/linux-3.10.0-1160.el7.x86_64/net/sctp/associola.c: 858
0xffffffffc0847508 <sctp_assoc_control_transport+520>:  mov    0x20(%rsi),%rax
/usr/src/debug/kernel-3.10.0-1160.el7/linux-3.10.0-1160.el7.x86_64/net/sctp/associola.c: 857
0xffffffffc084750c <sctp_assoc_control_transport+524>:  mov    %r10,%rdi
0xffffffffc084750f <sctp_assoc_control_transport+527>:  movslq 0xbc(%rax),%rdx

 852         /* Generate and send a SCTP_PEER_ADDR_CHANGE notification
 853          * to the user.
 854          */
 855         if (ulp_notify) {
 856                 memset(&addr, 0, sizeof(struct sockaddr_storage));
 857                 memcpy(&addr, &transport->ipaddr,
 858                        transport->af_specific->sockaddr_len);


The RAX, struct sctp_af is invalid.

crash> sctp_af.sockaddr_len -x
struct sctp_af {
  [0xbc] int sockaddr_len;
}

RAX: 3130636e6d2e736d
  • Another pattern:
[702302.031564] general protection fault: 0000 [#1] SMP
[702302.031592] Modules linked in: sctp bonding dm_mirror dm_region_hash dm_log dm_mod vfat fat skx_edac nfit libnvdimm intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm ipmi_ssif irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure sg mei_me mei lpc_ich hpilo hpwdt ipmi_si wmi ipmi_devintf ipmi_msghandler acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul uas crct10dif_common igb crc32c_intel smartpqi scsi_transport_sas ptp pps_core usb_storage dca i2c_algo_bit drm_panel_orientation_quirks
[702302.031839] CPU: 14 PID: 0 Comm: swapper/14 Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
[702302.031864] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 05/17/2022
[702302.031888] task: ffff97bab2f51080 ti: ffff97bab2f5c000 task.ti: ffff97bab2f5c000
[702302.031915] RIP: 0010:[<ffffffffc038f991>]  [<ffffffffc038f991>] sctp_do_8_2_transport_strike.isra.18+0x71/0x290 [sctp]
[702302.031952] RSP: 0018:ffff97c15f983b80  EFLAGS: 00010246
[702302.031968] RAX: 6d2e303130636e6d RBX: ffff97b96590fc00 RCX: 0000000000000001
[702302.031998] RDX: ce665c60626ac6c6 RSI: ffff97b96590fc00 RDI: ffff97baa4bb5000
[702302.032030] RBP: ffff97c15f983b98 R08: 0000000000000003 R09: ffff97c15f983c90
[702302.032050] R10: ffff97babfc03600 R11: 0000000000000005 R12: ffff97baa4bb5000
[702302.032071] R13: 0000000000000000 R14: 0000000000000003 R15: ffff97c15f983c90
[702302.032103] FS:  0000000000000000(0000) GS:ffff97c15f980000(0000) knlGS:0000000000000000
[702302.032125] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[702302.032141] CR2: 00007f066ee55ca0 CR3: 0000000eae410000 CR4: 00000000007607e0
[702302.032161] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[702302.032181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[702302.032201] PKRU: 00000000
[702302.032210] Call Trace:
[702302.032219]  <IRQ>
[702302.032233]  [<ffffffffc0391385>] sctp_cmd_interpreter.isra.22+0xf55/0x1450 [sctp]
[702302.032258]  [<ffffffffc038fc91>] sctp_do_sm+0xe1/0x350 [sctp]
[702302.032281]  [<ffffffffc03b1280>] ? sctp_oname+0x30/0x30 [sctp]
[702302.032301]  [<ffffffffc0390305>] sctp_generate_timeout_event+0xc5/0x110 [sctp]
[702302.032324]  [<ffffffffc03903d0>] ? sctp_generate_t4_rto_event+0x20/0x20 [sctp]
[702302.032347]  [<ffffffffc03903e3>] sctp_generate_t2_shutdown_event+0x13/0x20 [sctp]
[702302.032371]  [<ffffffff85aabd58>] call_timer_fn+0x38/0x110
[702302.032389]  [<ffffffffc03903d0>] ? sctp_generate_t4_rto_event+0x20/0x20 [sctp]
[702302.032411]  [<ffffffff85aae1ed>] run_timer_softirq+0x24d/0x300
[702302.032430]  [<ffffffff85aa4b95>] __do_softirq+0xf5/0x280
[702302.032448]  [<ffffffff861974ec>] call_softirq+0x1c/0x30
[702302.032466]  [<ffffffff85a2f715>] do_softirq+0x65/0xa0
[702302.032492]  [<ffffffff85aa4f15>] irq_exit+0x105/0x110
[702302.032518]  [<ffffffff86198a88>] smp_apic_timer_interrupt+0x48/0x60
[702302.032539]  [<ffffffff86194fba>] apic_timer_interrupt+0x16a/0x170
[702302.033257]  <EOI>
[702302.033267]  [<ffffffff85aaedab>] ? get_next_timer_interrupt+0xeb/0x260
[702302.034639]  [<ffffffff85fc7057>] ? cpuidle_enter_state+0x57/0xd0
[702302.035366]  [<ffffffff85fc71ae>] cpuidle_idle_call+0xde/0x230
[702302.036052]  [<ffffffff85a37c8e>] arch_cpu_idle+0xe/0xc0
[702302.036870]  [<ffffffff85b011ea>] cpu_startup_entry+0x14a/0x1e0
[702302.037935]  [<ffffffff85a5a7f7>] start_secondary+0x1f7/0x270
[702302.039014]  [<ffffffff85a000d5>] start_cpu+0x5/0x14
[702302.040088] Code: 09 83 f8 02 0f 84 70 01 00 00 85 c0 75 7c 45 85 ed 74 06 f6 43 24 02 74 35 48 8b 83 c0 00 00 00 48 8d 14 00 48 8b 83 b8 00 00 00 <48> 8b 80 68 02 00 00 48 39 c2 48 0f 46 c2 48 89 83 c0 00 00 00
[702302.041559] RIP  [<ffffffffc038f991>] sctp_do_8_2_transport_strike.isra.18+0x71/0x290 [sctp]
[702302.042265]  RSP <ffff97c15f983b80>

crash> dis -rl sctp_do_8_2_transport_strike+0x71|tail
/usr/src/debug/kernel-3.10.0-1160.el7/linux-3.10.0-1160.el7.x86_64/net/sctp/sm_sideeffect.c: 548
0xffffffffc0842974 <sctp_do_8_2_transport_strike+84>:   test   %r13d,%r13d
0xffffffffc0842977 <sctp_do_8_2_transport_strike+87>:   je     0xffffffffc084297f <sctp_do_8_2_transport_strike+95>
0xffffffffc0842979 <sctp_do_8_2_transport_strike+89>:   testb  $0x2,0x24(%rbx)
0xffffffffc084297d <sctp_do_8_2_transport_strike+93>:   je     0xffffffffc08429b4 <sctp_do_8_2_transport_strike+148>
/usr/src/debug/kernel-3.10.0-1160.el7/linux-3.10.0-1160.el7.x86_64/net/sctp/sm_sideeffect.c: 549
0xffffffffc084297f <sctp_do_8_2_transport_strike+95>:   mov    0xc0(%rbx),%rax
0xffffffffc0842986 <sctp_do_8_2_transport_strike+102>:  lea    (%rax,%rax,1),%rdx
0xffffffffc084298a <sctp_do_8_2_transport_strike+106>:  mov    0xb8(%rbx),%rax
0xffffffffc0842991 <sctp_do_8_2_transport_strike+113>:  mov    0x268(%rax),%rax

 548         if (!is_hb || transport->hb_sent) {
 549                 transport->rto = min((transport->rto * 2), transport->asoc->rto_max);

crash> sctp_transport -ox|grep -E "0xc0|0xb8"
   [0xb8] struct sctp_association *asoc;
   [0xc0] unsigned long rto;

RBX: ffff97b96590fc00
RAX: 6d2e303130636e6d

The RAX, struct sctp_association is invalid.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Updated: