[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 426632165 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003123] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009007] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011211] LSM: Security Framework initializing [ 0.012085] Yama: becoming mindful. [ 0.014053] SELinux: Initializing. [ 0.015111] *** VALIDATE selinux *** [ 0.024650] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030103] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031199] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032156] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033170] *** VALIDATE tmpfs *** [ 0.034584] *** VALIDATE proc *** [ 0.035344] *** VALIDATE cgroup *** [ 0.036017] *** VALIDATE cgroup2 *** [ 0.037372] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038207] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040047] Spectre V2 : User space: Vulnerable [ 0.041015] Speculative Store Bypass: Vulnerable [ 0.045053] debug: unmapping init [mem 0xffffffff93c59000-0xffffffff93c60fff] [ 0.048309] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050032] ... version: 2 [ 0.051000] ... bit width: 48 [ 0.052019] ... generic registers: 4 [ 0.053018] ... value mask: 0000ffffffffffff [ 0.054027] ... max period: 00007fffffffffff [ 0.055018] ... fixed-purpose events: 3 [ 0.056021] ... event mask: 000000070000000f [ 0.058308] rcu: Hierarchical SRCU implementation. [ 0.060804] smp: Bringing up secondary CPUs ... [ 0.061561] x86: Booting SMP configuration: [ 0.062028] .... node #0, CPUs: #1 #2 #3 [ 0.065511] smp: Brought up 1 node, 4 CPUs [ 0.067021] smpboot: Max logical packages: 1 [ 0.068034] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.251286] node 0 deferred pages initialised in 179ms [ 0.254320] devtmpfs: initialized [ 0.255365] x86/mm: Memory block size: 128MB [ 0.259123] gcov: version magic: 0x41383552 [ 0.262214] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.265109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.268347] pinctrl core: initialized pinctrl subsystem [ 0.270214] [ 0.270747] ************************************************************* [ 0.273016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275014] ** ** [ 0.277016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.280013] ** ** [ 0.282014] ** This means that this kernel is built to expose internal ** [ 0.284015] ** IOMMU data structures, which may compromise security on ** [ 0.287014] ** your system. ** [ 0.289012] ** ** [ 0.292015] ** If you see this message and you are not debugging the ** [ 0.294013] ** kernel, report this immediately to your vendor! ** [ 0.297015] ** ** [ 0.299028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.302016] ************************************************************* [ 0.305671] NET: Registered protocol family 16 [ 0.307455] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.310071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.312065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316129] cpuidle: using governor menu [ 0.317643] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.319469] PCI: Using configuration type 1 for base access [ 0.321127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.331042] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.333039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.336000] cryptd: max_cpu_qlen set to 1000 [ 0.338236] ACPI: Added _OSI(Module Device) [ 0.340018] ACPI: Added _OSI(Processor Device) [ 0.341013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.343014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.348868] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.353477] ACPI: Interpreter enabled [ 0.355063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.357016] ACPI: Using IOAPIC for interrupt routing [ 0.358115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.361427] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.371703] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.374047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.376019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.379083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.384382] acpiphp: Slot [2] registered [ 0.386143] acpiphp: Slot [5] registered [ 0.387120] acpiphp: Slot [6] registered [ 0.389141] acpiphp: Slot [3] registered [ 0.390105] acpiphp: Slot [4] registered [ 0.392101] acpiphp: Slot [7] registered [ 0.393125] acpiphp: Slot [8] registered [ 0.394104] acpiphp: Slot [9] registered [ 0.396108] acpiphp: Slot [10] registered [ 0.397117] acpiphp: Slot [11] registered [ 0.398113] acpiphp: Slot [12] registered [ 0.400130] acpiphp: Slot [13] registered [ 0.401097] acpiphp: Slot [14] registered [ 0.403126] acpiphp: Slot [15] registered [ 0.404147] acpiphp: Slot [16] registered [ 0.406103] acpiphp: Slot [17] registered [ 0.407132] acpiphp: Slot [18] registered [ 0.409182] acpiphp: Slot [19] registered [ 0.410143] acpiphp: Slot [20] registered [ 0.412110] acpiphp: Slot [21] registered [ 0.414110] acpiphp: Slot [22] registered [ 0.415100] acpiphp: Slot [23] registered [ 0.416117] acpiphp: Slot [24] registered [ 0.418107] acpiphp: Slot [25] registered [ 0.419119] acpiphp: Slot [26] registered [ 0.421118] acpiphp: Slot [27] registered [ 0.423112] acpiphp: Slot [28] registered [ 0.424109] acpiphp: Slot [29] registered [ 0.426142] acpiphp: Slot [30] registered [ 0.427157] acpiphp: Slot [31] registered [ 0.429072] PCI host bridge to bus 0000:00 [ 0.430028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.433030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.435030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.438035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.440028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.443034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.445185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.448051] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.452325] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.459597] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.463057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.465022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.467022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.470019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.473411] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475849] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.478048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.481863] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.486016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.494989] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.500017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.505710] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.523022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.538030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.575021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.587271] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.616046] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.622020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.635023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.646084] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.648376] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651428] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.654354] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.656243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.662114] iommu: Default domain type: Passthrough [ 0.664474] SCSI subsystem initialized [ 0.666341] ACPI: bus type USB registered [ 0.667104] usbcore: registered new interface driver usbfs [ 0.669089] usbcore: registered new interface driver hub [ 0.671090] usbcore: registered new device driver usb [ 0.673193] pps_core: LinuxPPS API ver. 1 registered [ 0.675013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.677250] PTP clock support registered [ 0.680045] EDAC MC: Ver: 3.0.0 [ 0.682051] PCI: Using ACPI for IRQ routing [ 0.683607] NetLabel: Initializing [ 0.685017] NetLabel: domain hash size = 128 [ 0.686010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.687303] NetLabel: unlabeled traffic allowed by default [ 0.690413] vgaarb: loaded [ 0.692314] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.694022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.701196] clocksource: Switched to clocksource kvm-clock [ 0.815844] VFS: Disk quotas dquot_6.6.0 [ 0.817586] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.820464] *** VALIDATE ramfs *** [ 0.821847] *** VALIDATE hugetlbfs *** [ 0.823924] pnp: PnP ACPI init [ 0.826384] pnp: PnP ACPI: found 6 devices [ 0.845223] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.847664] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.849205] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.850771] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.852565] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.854316] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.856573] NET: Registered protocol family 2 [ 0.858710] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.862947] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.866513] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.871703] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.874751] TCP: Hash tables configured (established 65536 bind 65536) [ 0.877817] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.881952] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885179] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.888778] NET: Registered protocol family 1 [ 0.891409] RPC: Registered named UNIX socket transport module. [ 0.893574] RPC: Registered udp transport module. [ 0.895547] RPC: Registered tcp transport module. [ 0.897514] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.899933] NET: Registered protocol family 44 [ 0.912088] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.914539] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916632] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.918931] PCI: CLS 0 bytes, default 64 [ 0.923701] Unpacking initramfs... [ 2.086378] debug: unmapping init [mem 0xffff9102fcc64000-0xffff9102fffcffff] [ 2.091912] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.093802] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.096402] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.672159] Initialise system trusted keyrings [ 2.674079] Key type blacklist registered [ 2.676119] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.685527] zbud: loaded [ 2.688184] *** VALIDATE nfs *** [ 2.689510] *** VALIDATE nfs4 *** [ 2.691275] pstore: using deflate compression [ 2.695358] Platform Keyring initialized [ 2.801838] NET: Registered protocol family 38 [ 2.803834] Key type asymmetric registered [ 2.805657] Asymmetric key parser 'x509' registered [ 2.807709] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.811240] io scheduler mq-deadline registered [ 2.812419] io scheduler kyber registered [ 2.813532] io scheduler bfq registered [ 2.815113] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.818196] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.820934] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.823551] ACPI: Power Button [PWRF] [ 2.828815] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.834656] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.844908] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.874278] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.903911] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.909718] Non-volatile memory driver v1.3 [ 2.911211] Linux agpgart interface v0.103 [ 2.949280] virtio_blk virtio1: [vda] 134104 512-byte logical blocks (68.7 MB/65.5 MiB) [ 2.952100] vda: detected capacity change from 0 to 68661248 [ 2.972826] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.976050] vdb: detected capacity change from 0 to 1073741824 [ 2.982630] libphy: Fixed MDIO Bus: probed [ 2.993644] usbcore: registered new interface driver usbserial_generic [ 2.996130] usbserial: USB Serial support registered for generic [ 2.998085] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.003120] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.005104] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.007542] mousedev: PS/2 mouse device common for all mice [ 3.011300] rtc_cmos 00:05: RTC can wake from S4 [ 3.014384] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.019565] rtc_cmos 00:05: registered as rtc0 [ 3.021236] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.022464] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.027486] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.029304] intel_pstate: CPU model not supported [ 3.036386] hid: raw HID events driver (C) Jiri Kosina [ 3.038855] usbcore: registered new interface driver usbhid [ 3.041065] usbhid: USB HID core driver [ 3.042699] drop_monitor: Initializing network drop monitor service [ 3.045345] Initializing XFRM netlink socket [ 3.047583] NET: Registered protocol family 10 [ 3.050799] Segment Routing with IPv6 [ 3.052517] NET: Registered protocol family 17 [ 3.056170] mpls_gso: MPLS GSO support [ 3.062915] RAS: Correctable Errors collector initialized. [ 3.065138] AVX version of gcm_enc/dec engaged. [ 3.066899] AES CTR mode by8 optimization enabled [ 3.144779] sched_clock: Marking stable (3144747711, 0)->(4123820209, -979072498) [ 3.148929] registered taskstats version 1 [ 3.150992] Loading compiled-in X.509 certificates [ 3.153314] zswap: loaded using pool lzo/zbud [ 3.178733] Key type big_key registered [ 3.191363] Key type encrypted registered [ 3.193113] ima: No TPM chip found, activating TPM-bypass! [ 3.195021] ima: Allocated hash algorithm: sha1 [ 3.196656] ima: No architecture policies found [ 3.198441] evm: Initialising EVM extended attributes: [ 3.200616] evm: security.selinux [ 3.201714] evm: security.ima [ 3.202577] evm: security.capability [ 3.203891] evm: HMAC attrs: 0x1 [ 3.206583] rtc_cmos 00:05: setting system clock to 2026-01-16 09:10:30 UTC (1768554630) [ 3.212777] debug: unmapping init [mem 0xffffffff94c03000-0xffffffff94dfffff] [ 3.216032] debug: unmapping init [mem 0xffffffff93982000-0xffffffff93c58fff] [ 3.225686] Write protecting the kernel read-only data: 28672k [ 3.227839] debug: unmapping init [mem 0xffffffff92003000-0xffffffff921fffff] [ 3.230731] debug: unmapping init [mem 0xffffffff92914000-0xffffffff929fffff] [ 3.258715] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.267864] systemd[1]: Detected virtualization kvm. [ 3.269663] systemd[1]: Detected architecture x86-64. [ 3.271334] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.293845] systemd[1]: No hostname configured. [ 3.295770] systemd[1]: Set hostname to . [ 3.298102] random: systemd: uninitialized urandom read (16 bytes read) [ 3.300806] systemd[1]: Initializing machine ID from random generator. [ 3.349935] random: ln: uninitialized urandom read (6 bytes read) [ 3.448579] random: systemd: uninitialized urandom read (16 bytes read) [ 3.451307] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.456484] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.459729] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.131418] device-mapper: uevent: version 1.0.3 [ 4.132887] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.955385] virtio_net virtio0 ens2: renamed from eth0 [ 5.135287] scsi host0: ata_piix [ 5.154320] scsi host1: ata_piix [ 5.155858] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.158251] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.413751] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.713769] random: crng init done [ 9.715332] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.202904] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.446630] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.709095] SELinux: Disabled at runtime. [ 11.769607] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.777955] systemd[1]: Detected virtualization kvm. [ 11.779924] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.259762] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.263433] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.271594] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.275685] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.279320] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.286936] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.291618] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ 12.347908] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 12.750893] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.173039] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.213239] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.266198] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.284463] EDAC sbridge: Ver: 1.1.2 [ 14.471201] Key type dns_resolver registered [ 14.768900] NFS: Registering the id_resolver key type [ 14.770917] Key type id_resolver registered [ 14.772456] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg217-client login: [ 44.318115] libcfs: loading out-of-tree module taints kernel. [ 44.409847] Key type ._llcrypt registered [ 44.411987] Key type .llcrypt registered [ 44.604652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.613060] alg: No test for adler32 (adler32-zlib) [ 45.635219] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 45.991316] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 47.623186] Key type lgssc registered [ 48.282969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 176.545851] Lustre: Mounted lustre-client [ 181.930910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 197.953397] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing check_logdir /tmp/testlogs/ [ 202.208057] Lustre: lustre-OST0000-osc-ffff9103497d7000: disconnect after 23s idle [ 204.488898] Lustre: DEBUG MARKER: oleg217-client.virtnet: executing yml_node [ 210.164995] Lustre: DEBUG MARKER: Client: 2.17.50.30 [ 213.070981] Lustre: DEBUG MARKER: MDS: 2.17.50.30 [ 215.759465] Lustre: DEBUG MARKER: OSS: 2.17.50.30 [ 217.191837] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jan 16 04:14:03 EST 2026 [ 235.723740] LustreError: 7904:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9103497d7000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 235.735568] LustreError: 7904:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 235.770345] Lustre: Unmounted lustre-client [ 273.954629] Key type lgssc unregistered [ 274.210699] LNet: 8352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.216454] LNetError: 8352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 274.247475] LNet: Removed LNI 192.168.202.17@tcp [ 275.018163] Key type .llcrypt unregistered [ 275.019680] Key type ._llcrypt unregistered [ 286.372016] hrtimer: interrupt took 5980803 ns [ 303.940489] Key type ._llcrypt registered [ 303.943549] Key type .llcrypt registered [ 304.231793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 304.251710] alg: No test for adler32 (adler32-zlib) [ 305.176716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 305.235338] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 305.271793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 309.602933] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 309.928616] LNet: 9227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.935261] LNetError: 9227:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.946380] LNet: Removed LNI 192.168.202.17@tcp [ 310.790660] Key type .llcrypt unregistered [ 310.794721] Key type ._llcrypt unregistered [ 317.769370] Key type ._llcrypt registered [ 317.771365] Key type .llcrypt registered [ 319.335437] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:15:44 (1768554944) [ 319.652965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 319.671601] alg: No test for adler32 (adler32-zlib) [ 320.586582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.725953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.766447] LNet: 9634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.809201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.878878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 320.907637] LNet: 9641:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 327.126561] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:15:53 (1768554953) [ 327.482177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.500402] LNet: 10218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.515961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.600872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 334.686784] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:16:00 (1768554960) [ 334.984815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.010198] LNet: 10804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.044204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.153923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 341.594735] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:16:07 (1768554967) [ 341.712348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.754029] LNet: 11383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.821809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.947499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 349.395831] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:16:15 (1768554975) [ 349.584215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.613527] LNet: 11973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.668694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.756308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 350.321745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 357.303712] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:16:23 (1768554983) [ 357.728606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.756547] LNet: 12565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.778156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.864684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 358.329377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 358.865695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 366.313876] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:16:31 (1768554991) [ 366.586641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.645339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.743480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 373.918787] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:16:39 (1768554999) [ 374.042661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.082081] LNet: 13743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.087099] LNet: 13743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 374.153134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.200923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 374.219690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 374.729640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 374.786900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 375.298512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 375.324973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.780481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 375.801732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 376.207610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 376.254343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.735505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 376.774263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 377.190660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 377.222557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 377.511371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 377.538536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.792832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 377.827795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 378.083593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 378.113512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.466549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 378.516795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.958376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 379.018537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.402526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 379.490015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 386.826855] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:16:52 (1768555012) [ 386.929743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.968461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.061992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 387.106384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 393.509419] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:16:59 (1768555019) [ 393.614089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.676076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.808855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 393.857746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 400.049990] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:17:06 (1768555026) [ 400.185031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.268234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.405793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 400.448997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 407.231872] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:17:13 (1768555033) [ 407.546627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.575254] LNet: 16188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.584482] LNet: 16188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 407.625707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.751677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 407.799865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 413.498969] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:17:19 (1768555039) [ 413.864785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.921951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.029815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 414.088877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 420.118676] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:17:26 (1768555046) [ 420.358988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.399923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.453701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 420.487611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 428.140853] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:17:33 (1768555053) [ 428.262812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.335959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.521081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.324679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 437.036369] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:17:43 (1768555063) [ 437.261855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.289525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.494891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.766194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.240871] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:17:50 (1768555070) [ 444.535040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.560767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.616208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.731819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 451.782878] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:17:57 (1768555077) [ 451.894238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.935048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.471325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.659284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.263557] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:18:05 (1768555085) [ 459.535192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.592638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.671338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.752192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.850805] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:18:12 (1768555092) [ 466.125683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.180309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.263947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.323351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.332582] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:18:18 (1768555098) [ 472.712665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.750694] LNet: 21629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.756243] LNet: 21629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 472.779956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.882646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 479.178655] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:18:25 (1768555105) [ 479.574956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.633777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.765561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 487.020392] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:18:32 (1768555112) [ 487.399151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.451776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.549466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 488.240972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 497.855196] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:18:43 (1768555123) [ 498.258824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.357771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.540746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 498.664728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 507.966110] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:18:53 (1768555133) [ 508.094713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.166658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.286532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 508.314946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 517.097890] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:19:02 (1768555142) [ 517.423460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.484450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.553238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 517.589747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 525.633082] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:19:11 (1768555151) [ 525.747408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.828505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.962216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 526.011487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 534.718563] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:19:20 (1768555160) [ 535.097133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.129138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.235363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 535.405552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 535.461992] LNetError: 25748:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 535.589804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 535.666876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 542.069989] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:19:28 (1768555168) [ 542.336763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.464900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.576494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PS8Q7HIlLD [ 542.677624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PS8Q7HIlLD [ 542.794408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PS8Q7HIlLD [ 550.767256] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:19:36 (1768555176) [ 551.048428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.082722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.140981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 551.168095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 551.224484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 558.092269] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:19:43 (1768555183) [ 558.198819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.286587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.378262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 561.379845] LNet: Message forwarding will be enabled in 2s [ 564.447143] LNet: Message forwarding enabled [ 564.857142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 564.875632] LNet: Message forwarding will stop in 2s to 6s [ 569.407845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 570.513447] LNet: Message forwarding disabled [ 572.092911] LNet: Message forwarding will be enabled in 2s [ 575.647670] LNet: Message forwarding enabled [ 579.193634] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:20:05 (1768555205) [ 579.580480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.562280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.627325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 587.705622] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:20:13 (1768555213) [ 587.875471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.962158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.082228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 588.113534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 588.150379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 588.187455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 588.237681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 588.279549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 588.345814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 588.385121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 588.447463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 588.490777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 588.527059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 588.580848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 588.635510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 588.704894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 588.746280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 588.791517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 588.850931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 588.900631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 588.939928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 588.968905] LNetError: 28821:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 596.572497] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:20:22 (1768555222) [ 596.745851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.823108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.904686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.511217] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:20:29 (1768555229) [ 606.577496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg217-server [ 608.421370] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:20:34 (1768555234) [ 608.638669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.661967] LNet: 30323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.668776] LNet: 30323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 608.686938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.727287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 608.749487] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 610.144492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[18-22]@tcp [ 610.163827] LNet: 30537:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 610.227639] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.18@tcp is being used as a gateway but routing feature is not turned on [ 610.243246] LNetError: 30327:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.18@tcp has gone from up to down [ 611.235738] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.20@tcp is being used as a gateway but routing feature is not turned on [ 611.258412] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 612.273129] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.20@tcp is being used as a gateway but routing feature is not turned on [ 612.291447] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 614.346840] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.18@tcp is being used as a gateway but routing feature is not turned on [ 614.372889] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 615.955807] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:20:41 (1768555241) [ 618.238628] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg217-server [ 618.474218] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.21@tcp is being used as a gateway but routing feature is not turned on [ 618.498960] LNetError: 30327:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 620.697658] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:20:46 (1768555246) [ 620.852742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.877949] LNetError: 31465:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.895030] LNet: Removed LNI 192.168.202.17@tcp [ 620.928511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.035811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 621.059827] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 622.072302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[18-23/2]@tcp [ 622.109969] LNet: 31590:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 622.116951] LNet: 31590:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 622.166852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[18-23/2]@tcp [ 622.167724] LNetError: 31475:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.22@tcp has gone from up to down [ 622.207836] LNetError: 31475:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 629.350728] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:20:55 (1768555255) [ 629.455362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.504037] LNetError: 32174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.569812] LNet: Removed LNI 192.168.202.17@tcp [ 630.589068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.698482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 630.752986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.794628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.855695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.887847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.907218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.948289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 631.019558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.052733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.114836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.194262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.227993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.262993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 631.327172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.354516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.400573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.460690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.493962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.532843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 631.572414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.590889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.639312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.682755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.715931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.756570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 631.810865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.827296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.881248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.940953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.968750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.011937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 632.059477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.091784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.147460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.702968] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:21:03 (1768555263) [ 639.112536] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg217-server [ 640.549582] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:21:06 (1768555266) [ 642.129836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg217-server [ 644.142452] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:21:10 (1768555270) [ 645.171155] Key type .llcrypt unregistered [ 645.174906] Key type ._llcrypt unregistered [ 652.304960] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 652.495564] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 652.504889] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 652.527326] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 655.357702] Key type ._llcrypt registered [ 655.359825] Key type .llcrypt registered [ 655.535884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 655.547493] alg: No test for adler32 (adler32-zlib) [ 656.469323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.537304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.579128] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 657.016341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 657.037084] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 658.723424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 658.743211] LNet: Removed LNI 10.1.2.3@tcp [ 658.903357] LNet: 34299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.908673] LNetError: 34299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.919906] LNet: Removed LNI 192.168.202.17@tcp [ 659.485546] Key type .llcrypt unregistered [ 659.490902] Key type ._llcrypt unregistered [ 671.043795] Key type ._llcrypt registered [ 671.045410] Key type .llcrypt registered [ 672.573074] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:21:38 (1768555298) [ 673.306625] Key type .llcrypt unregistered [ 673.309175] Key type ._llcrypt unregistered [ 680.017244] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 680.176277] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 680.184381] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 680.189387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 682.753346] Key type ._llcrypt registered [ 682.755974] Key type .llcrypt registered [ 682.954453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 682.964566] alg: No test for adler32 (adler32-zlib) [ 683.860203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.941864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 683.962775] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 684.979748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 685.003616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 687.692920] LNet: 35787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.700441] LNetError: 35787:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.708208] LNet: Removed LNI 192.168.202.17@tcp [ 687.718465] LNet: Removed LNI 10.1.2.3@tcp [ 688.312333] Key type .llcrypt unregistered [ 688.314415] Key type ._llcrypt unregistered [ 703.505499] Key type ._llcrypt registered [ 703.511571] Key type .llcrypt registered [ 704.963354] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:22:10 (1768555330) [ 705.752409] Key type .llcrypt unregistered [ 705.755683] Key type ._llcrypt unregistered [ 712.982379] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 713.383600] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 713.389707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 713.396193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.463774] Key type ._llcrypt registered [ 716.465562] Key type .llcrypt registered [ 716.781985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.822904] alg: No test for adler32 (adler32-zlib) [ 717.858517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.922393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 717.960919] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 720.236355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 720.259212] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 722.560207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 722.592641] LNet: Removed LNI 10.1.2.3@tcp [ 722.612826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 722.662182] LNet: Removed LNI 10.1.2.31@tcp [ 722.687483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 722.712671] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 723.474628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 723.496618] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 724.316251] LNet: 37702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.321612] LNetError: 37702:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 724.327287] LNet: Removed LNI 10.1.2.3@tcp [ 724.337424] LNet: Removed LNI 10.1.2.31@tcp [ 725.065756] Key type .llcrypt unregistered [ 725.068787] Key type ._llcrypt unregistered [ 738.770442] Key type ._llcrypt registered [ 738.771721] Key type .llcrypt registered [ 740.326883] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:22:45 (1768555365) [ 740.910255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 740.922952] alg: No test for adler32 (adler32-zlib) [ 741.991691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.074597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.122563] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 743.284919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.316595] LNet: 38653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.326817] LNetError: 38653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.338784] LNet: Removed LNI 192.168.202.17@tcp [ 743.365370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.446585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.17@tcp [ 743.474601] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 744.400254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.17@tcp [ 744.434678] LNet: Removed LNI 192.168.202.17@tcp [ 744.526873] LNet: 38788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.189487] Key type .llcrypt unregistered [ 745.195222] Key type ._llcrypt unregistered [ 756.373936] Key type ._llcrypt registered [ 756.376699] Key type .llcrypt registered [ 757.449546] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:23:03 (1768555383) [ 759.401947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg217-server [ 761.528456] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:23:07 (1768555387) [ 762.119435] Key type .llcrypt unregistered [ 762.124247] Key type ._llcrypt unregistered [ 768.277570] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 768.494025] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 768.504669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 768.542746] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 769.965637] Key type ._llcrypt registered [ 769.970398] Key type .llcrypt registered [ 770.212783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 770.226524] alg: No test for adler32 (adler32-zlib) [ 771.180989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.260385] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 771.876608] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 773.745187] LNet: 40610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.749276] LNetError: 40610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.753751] LNet: Removed LNI 192.168.202.17@tcp1 [ 773.760872] LNet: Removed LNI 10.1.2.3@tcp2 [ 774.408437] Key type .llcrypt unregistered [ 774.415129] Key type ._llcrypt unregistered [ 786.247915] Key type ._llcrypt registered [ 786.250651] Key type .llcrypt registered [ 787.647720] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:23:33 (1768555413) [ 788.391820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.415864] alg: No test for adler32 (adler32-zlib) [ 789.331654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.388729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.415168] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 790.422784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.461850] LNet: 41575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.468342] LNetError: 41575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.477699] LNet: Removed LNI 192.168.202.17@tcp [ 790.493931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.559141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 790.597341] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 800.372774] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:23:45 (1768555425) [ 800.575937] LNet: 42158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.589718] LNetError: 42158:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.626826] LNet: Removed LNI 192.168.202.17@tcp [ 801.686683] Key type .llcrypt unregistered [ 801.689265] Key type ._llcrypt unregistered [ 809.374362] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 809.609985] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 809.618785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 809.632627] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 812.465672] Key type ._llcrypt registered [ 812.468142] Key type .llcrypt registered [ 812.703661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.732483] alg: No test for adler32 (adler32-zlib) [ 813.616331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.689269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 813.721307] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 814.578587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 814.633036] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.088098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.107405] LNet: 42905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.115440] LNetError: 42905:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.123885] LNet: Removed LNI 192.168.202.17@tcp [ 817.141403] LNet: Removed LNI 10.1.2.3@tcp [ 817.174687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.251855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 817.268870] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 817.273223] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.680262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.709660] LNet: 42932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.724492] LNetError: 42932:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.727542] LNetError: 42932:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 817.736043] LNet: Removed LNI 192.168.202.17@tcp [ 817.746934] LNet: Removed LNI 10.1.2.3@tcp [ 817.775408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.842631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 817.874308] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 817.881713] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.275683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 818.294369] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 819.216961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.237057] LNet: 43094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.241835] LNetError: 43094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.249987] LNetError: 43094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 819.278073] LNet: Removed LNI 192.168.202.17@tcp [ 819.289537] LNet: Removed LNI 10.1.2.3@tcp [ 819.307561] LNet: Removed LNI 192.168.202.17@tcp2 [ 819.357175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.421460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 819.447135] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 819.452763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 819.458228] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 827.709806] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:24:13 (1768555453) [ 828.098861] LNet: 43693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.105123] LNetError: 43693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.109940] LNetError: 43693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 828.115789] LNet: Removed LNI 192.168.202.17@tcp [ 828.118348] LNet: Removed LNI 10.1.2.3@tcp [ 828.129552] LNet: Removed LNI 192.168.202.17@tcp2 [ 828.692462] Key type .llcrypt unregistered [ 828.695950] Key type ._llcrypt unregistered [ 836.607725] Key type ._llcrypt registered [ 836.609979] Key type .llcrypt registered [ 836.893360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 836.934732] alg: No test for adler32 (adler32-zlib) [ 837.954565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.030408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 838.062544] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 839.040765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.063582] LNet: 44084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.069299] LNetError: 44084:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.082165] LNet: Removed LNI 192.168.202.17@tcp [ 839.100406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.157431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 839.185776] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 839.758401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.787472] LNet: 44114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.798965] LNetError: 44114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.818081] LNet: Removed LNI 192.168.202.17@tcp [ 839.838111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.903050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 839.933987] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 840.600335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.634410] LNet: Removed LNI 192.168.202.17@tcp [ 840.649408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.728681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.749076] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 841.182076] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 841.330114] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 841.336434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 841.339359] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.658687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.684367] LNet: 44218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.692021] LNet: 44218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 843.701883] LNetError: 44218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.707547] LNetError: 44218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 843.735240] LNet: Removed LNI 192.168.202.17@tcp [ 843.758677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.801417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 843.823451] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 850.450304] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:24:36 (1768555476) [ 850.516157] LNet: 44806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.523719] LNetError: 44806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.548097] LNet: Removed LNI 192.168.202.17@tcp [ 851.124502] Key type .llcrypt unregistered [ 851.126975] Key type ._llcrypt unregistered [ 856.933659] Key type ._llcrypt registered [ 856.936105] Key type .llcrypt registered [ 857.059828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.068301] alg: No test for adler32 (adler32-zlib) [ 857.975442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.019188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.041841] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 858.631240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.651212] LNet: 45209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.662591] LNetError: 45209:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.675233] LNet: Removed LNI 192.168.202.17@tcp [ 858.705921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.763094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 858.780726] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 865.262984] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:24:51 (1768555491) [ 865.548953] LNet: 45801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.552968] LNetError: 45801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.576639] LNet: Removed LNI 192.168.202.17@tcp [ 866.104774] Key type .llcrypt unregistered [ 866.107105] Key type ._llcrypt unregistered [ 872.868940] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 873.005688] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.008120] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 873.042365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 875.895297] Key type ._llcrypt registered [ 875.899846] Key type .llcrypt registered [ 876.194380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.226426] alg: No test for adler32 (adler32-zlib) [ 877.258686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.364510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.413384] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 878.911901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.939368] LNet: 46247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.952323] LNetError: 46247:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.979965] LNet: Removed LNI 192.168.202.17@tcp [ 879.016889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.121318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 879.186624] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 888.172225] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:25:13 (1768555513) [ 888.379452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.427370] LNet: 46845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.443374] LNetError: 46845:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.464416] LNet: Removed LNI 192.168.202.17@tcp [ 888.483370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.636444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 896.901749] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:25:22 (1768555522) [ 897.067517] LNet: 47425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.884108] Key type .llcrypt unregistered [ 897.885790] Key type ._llcrypt unregistered [ 904.594440] Key type ._llcrypt registered [ 904.596541] Key type .llcrypt registered [ 904.777418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.790636] alg: No test for adler32 (adler32-zlib) [ 905.711523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.742866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 905.760349] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 906.440514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.466225] LNet: 47821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.471729] LNetError: 47821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.481835] LNet: Removed LNI 192.168.202.17@tcp [ 906.499496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.563533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 906.587789] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 914.037548] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:25:39 (1768555539) [ 914.262464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.285402] LNet: 48416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.291132] LNetError: 48416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.308441] LNet: Removed LNI 192.168.202.17@tcp [ 914.328335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.443450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 920.876609] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:25:46 (1768555546) [ 922.520660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 924.449106] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:25:50 (1768555550) [ 926.527645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 928.684498] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:25:54 (1768555554) [ 930.548344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 932.291449] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:25:58 (1768555558) [ 933.447785] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 935.556405] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:26:01 (1768555561) [ 937.296795] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 938.821128] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:26:04 (1768555564) [ 939.107419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.130416] LNet: 50469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.149494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.198897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 939.217384] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 939.617959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.17 [ 939.637517] LNet: 50602:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 939.650027] LNet: Removed LNI 192.168.202.17@tcp [ 945.928378] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:26:11 (1768555571) [ 946.218435] LNet: 51173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.228515] LNetError: 51173:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.243053] LNet: Removed LNI 192.168.202.17@tcp [ 946.928488] Key type .llcrypt unregistered [ 946.930932] Key type ._llcrypt unregistered [ 953.510618] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.721353] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.724886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.732291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 956.473535] Key type ._llcrypt registered [ 956.476275] Key type .llcrypt registered [ 956.601862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.619237] alg: No test for adler32 (adler32-zlib) [ 957.995637] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.035863] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:26:30 (1768555590) [ 965.246310] LNet: 52070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.257789] LNetError: 52070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.312667] LNet: Removed LNI 10.1.2.3@tcp [ 966.073626] Key type .llcrypt unregistered [ 966.075263] Key type ._llcrypt unregistered [ 973.571840] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 973.808648] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 973.815973] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 973.820540] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 976.656252] Key type ._llcrypt registered [ 976.658267] Key type .llcrypt registered [ 976.867984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.892668] alg: No test for adler32 (adler32-zlib) [ 977.825339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 984.901393] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:26:50 (1768555610) [ 985.298235] LNet: 52963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.307987] LNetError: 52963:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 985.328334] LNet: Removed LNI 10.1.2.3@tcp [ 985.895090] Key type .llcrypt unregistered [ 985.904490] Key type ._llcrypt unregistered [ 993.346492] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.619720] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.631212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 993.634640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 995.470905] Key type ._llcrypt registered [ 995.473751] Key type .llcrypt registered [ 995.624379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 995.635231] alg: No test for adler32 (adler32-zlib) [ 996.621418] LNet: 53251:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 996.637659] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.883715] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:27:09 (1768555629) [ 1003.182446] LNet: 53844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.196803] LNetError: 53844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.219167] LNet: Removed LNI 10.1.2.3@tcp [ 1003.704675] Key type .llcrypt unregistered [ 1003.706696] Key type ._llcrypt unregistered [ 1009.961976] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1010.329810] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1010.331936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1010.360410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1013.054165] Key type ._llcrypt registered [ 1013.055976] Key type .llcrypt registered [ 1013.263089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1013.283943] alg: No test for adler32 (adler32-zlib) [ 1014.161621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.209033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1022.090587] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:27:28 (1768555648) [ 1022.697864] LNet: 55069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.703378] LNetError: 55069:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1022.717945] LNet: Removed LNI 10.1.2.3@tcp [ 1023.263516] Key type .llcrypt unregistered [ 1023.265124] Key type ._llcrypt unregistered [ 1032.007445] Key type ._llcrypt registered [ 1032.011816] Key type .llcrypt registered [ 1032.228499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.283650] alg: No test for adler32 (adler32-zlib) [ 1033.343186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1033.398805] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1045.550596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1046.661281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_interrupt -r 1 [ 1046.690636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_interrupt -r 1 [ 1046.757507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1046.784558] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1047.155666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1047.232440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1047.300955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1047.356295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.397328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.448222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.520336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.563637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.605162] LNet: 55509:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.653750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.717645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.777399] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1047.779664] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1047.796341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.844155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.883906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.926112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.982320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1048.608344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_dropped -r 1 [ 1048.648535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_dropped -r 1 [ 1048.695215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1049.012430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.069594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1049.142824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1049.214287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1049.265739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1049.364442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1049.408625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1049.479066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1049.503476] LNet: 55618:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1049.528888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1049.600641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1049.677842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.744463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.795895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.824536] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1049.828317] LNet: Skipped 1 previous similar message [ 1049.837996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.867924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1050.414813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_aborted -r 1 [ 1050.452875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_aborted -r 1 [ 1050.534095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1050.837408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.876472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1050.904970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.929574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.962644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.987984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.011581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.042430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.060361] LNet: 55728:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.086226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.134477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.166581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.195186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.226662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.267915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.332208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1051.851898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_no_route -r 1 [ 1051.880165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_no_route -r 1 [ 1051.904974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1051.921316] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1051.928986] LNet: Skipped 1 previous similar message [ 1052.128876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.151670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1052.167084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.204411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.223413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.263944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.284331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.315784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.340570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.372097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.393860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.421264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.457787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.496442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.518089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1052.850202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_timeout -r 1 [ 1052.915549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_timeout -r 1 [ 1053.001142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1053.456531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.524431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1053.553442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.582416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.609646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.637252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.665614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.694261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.713453] LNet: 55946:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.720985] LNet: 55946:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1053.743968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.786503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.826488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.861180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.899878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.944474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.975251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1054.747904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_error -r 1 [ 1054.836800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_error -r 1 [ 1054.885316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1055.313463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.383120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1055.458338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.532357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.630951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.695495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.764085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.811523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.857380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.917275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.960761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.017972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.087145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.144322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.210561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.536481] LNet: 56095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.549033] LNetError: 56095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.582226] LNet: Removed LNI 192.168.202.17@tcp [ 1057.194525] Key type .llcrypt unregistered [ 1057.196216] Key type ._llcrypt unregistered [ 1080.509515] Key type ._llcrypt registered [ 1080.512219] Key type .llcrypt registered [ 1082.075330] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:28:27 (1768555707) [ 1083.099433] Key type .llcrypt unregistered [ 1083.103680] Key type ._llcrypt unregistered [ 1090.434839] Key type ._llcrypt registered [ 1090.441609] Key type .llcrypt registered [ 1090.677331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.690772] alg: No test for adler32 (adler32-zlib) [ 1091.604974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.629932] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1100.201292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1112.364498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.408493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1112.443048] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1113.729512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_interrupt -r 1 [ 1113.774667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_interrupt -r 1 [ 1113.816560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_interrupt -r 1 [ 1113.867827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_interrupt -r 1 [ 1113.899703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_interrupt -r 1 [ 1113.931878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_interrupt -r 1 [ 1113.970952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_interrupt -r 1 [ 1114.008548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_interrupt -r 1 [ 1114.037505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1114.060637] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1114.069036] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1114.275248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.297623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1114.361453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.423477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.474547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.540295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.576834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.612489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.630570] LNet: 57559:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.652489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.696297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.720140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.758579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.799311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.828894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.850786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1115.248375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_dropped -r 1 [ 1115.303791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_dropped -r 1 [ 1115.358308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_dropped -r 1 [ 1115.406683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_dropped -r 1 [ 1115.456669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_dropped -r 1 [ 1115.490564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_dropped -r 1 [ 1115.556762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_dropped -r 1 [ 1115.617644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_dropped -r 1 [ 1115.680587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1115.704939] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1115.708379] LNet: Skipped 2 previous similar messages [ 1115.929040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.974361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.007309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.047507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.079365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.117181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.171559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.208333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.239562] LNet: 57681:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.256988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.300549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.339248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.394974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.427644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.455897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.489045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.007583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_aborted -r 1 [ 1117.040098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_aborted -r 1 [ 1117.067659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_aborted -r 1 [ 1117.097871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_aborted -r 1 [ 1117.126894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_aborted -r 1 [ 1117.171448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_aborted -r 1 [ 1117.202166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_aborted -r 1 [ 1117.236716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_aborted -r 1 [ 1117.266769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1117.297714] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1117.303316] LNet: Skipped 2 previous similar messages [ 1117.548210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.568525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1117.591733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.616450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.638790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.664755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.691743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.730979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.745907] LNet: 57802:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1117.765853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.796054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.824421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.855251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.890306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.935852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.991966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.541824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_no_route -r 1 [ 1118.595139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_no_route -r 1 [ 1118.628653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_no_route -r 1 [ 1118.657833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_no_route -r 1 [ 1118.725186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_no_route -r 1 [ 1118.776661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_no_route -r 1 [ 1118.817737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_no_route -r 1 [ 1118.855845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_no_route -r 1 [ 1118.886697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1119.118720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.157834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1119.194532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.231158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.272136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.315943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.367833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.406442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.437882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.476181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.495981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.513708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.534415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.570703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.602073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1120.026860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_timeout -r 1 [ 1120.069895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_timeout -r 1 [ 1120.107277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_timeout -r 1 [ 1120.144962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_timeout -r 1 [ 1120.189776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_timeout -r 1 [ 1120.238569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_timeout -r 1 [ 1120.310768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_timeout -r 1 [ 1120.362942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_timeout -r 1 [ 1120.416358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1120.462132] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1120.471096] LNet: Skipped 5 previous similar messages [ 1120.784863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.813305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1120.852205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.887779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.912959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.942614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.963201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.001767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.021060] LNet: 58044:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.026558] LNet: 58044:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1121.040252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.064181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.088748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.111760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.140625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.166187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.208840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1121.613745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_error -r 1 [ 1121.642514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_error -r 1 [ 1121.671593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_error -r 1 [ 1121.696818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_error -r 1 [ 1121.740389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_error -r 1 [ 1121.771982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_error -r 1 [ 1121.800815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_error -r 1 [ 1121.828965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_error -r 1 [ 1121.853821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1122.036889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.062399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1122.083836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.103835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.127482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.150320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.176675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.202633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.223823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.246799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.270465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.292320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.313931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.343670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.368401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1128.067786] LNet: 58240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.074718] LNetError: 58240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.081625] LNet: Removed LNI 192.168.202.17@tcp [ 1128.094550] LNet: Removed LNI 192.168.202.17@tcp1 [ 1128.515732] Key type .llcrypt unregistered [ 1128.517896] Key type ._llcrypt unregistered [ 1146.038503] Key type ._llcrypt registered [ 1146.045759] Key type .llcrypt registered [ 1147.442224] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:29:33 (1768555773) [ 1148.596446] Key type .llcrypt unregistered [ 1148.598244] Key type ._llcrypt unregistered [ 1156.758454] Key type ._llcrypt registered [ 1156.759963] Key type .llcrypt registered [ 1157.206644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1157.262914] alg: No test for adler32 (adler32-zlib) [ 1158.183677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1158.234711] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1166.717269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1167.681914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e remote_dropped -r 1 [ 1167.732296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e remote_dropped -r 1 [ 1167.822958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1167.852707] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1168.187949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.248453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1168.342334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.425895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.468474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.542648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.588227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.676453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.702817] LNet: 59483:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.735809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.784409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.834839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.873709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.906401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.942401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.977748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.713560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e remote_error -r 1 [ 1169.787605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e remote_error -r 1 [ 1169.860567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1169.909168] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1170.148564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.199739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1170.232628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.287250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.323826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.373203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.451561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.513942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.537194] LNet: 59592:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.582105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.663083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.764255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.816431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.869248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.961910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.018414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1171.601463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e remote_timeout -r 1 [ 1171.635530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e remote_timeout -r 1 [ 1171.669935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1171.683310] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1171.892595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.926833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1171.958917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.013677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.042754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.065363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.090781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.112889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.125623] LNet: 59702:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.147794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.184796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.217314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.243669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.280277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.310231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.339685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.625408] LNet: 59742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.630812] LNetError: 59742:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.641863] LNet: Removed LNI 192.168.202.17@tcp [ 1172.984636] Key type .llcrypt unregistered [ 1172.987989] Key type ._llcrypt unregistered [ 1190.798819] Key type ._llcrypt registered [ 1190.803237] Key type .llcrypt registered [ 1192.112732] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:30:17 (1768555817) [ 1193.138431] Key type .llcrypt unregistered [ 1193.141919] Key type ._llcrypt unregistered [ 1199.048310] Key type ._llcrypt registered [ 1199.051141] Key type .llcrypt registered [ 1199.291218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.320374] alg: No test for adler32 (adler32-zlib) [ 1200.243565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1200.269109] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1207.656983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1216.492567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1216.525573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1216.540622] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1217.807393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e remote_dropped -r 1 [ 1217.838938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e remote_dropped -r 1 [ 1217.863699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e remote_dropped -r 1 [ 1217.885861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e remote_dropped -r 1 [ 1217.911771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e remote_dropped -r 1 [ 1217.939471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e remote_dropped -r 1 [ 1217.964819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e remote_dropped -r 1 [ 1218.000694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e remote_dropped -r 1 [ 1218.050559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1218.072887] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1218.082625] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.117@tcp [ 1218.328855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.352419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1218.378799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.417474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.458099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.501558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.571767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.620848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.650539] LNet: 61202:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.676315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.717720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.811677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.869793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.924201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.968036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.017607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1219.049461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.451575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e remote_error -r 1 [ 1219.488533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e remote_error -r 1 [ 1219.528263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e remote_error -r 1 [ 1219.571498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e remote_error -r 1 [ 1219.621086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e remote_error -r 1 [ 1219.656321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e remote_error -r 1 [ 1219.694176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e remote_error -r 1 [ 1219.727603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e remote_error -r 1 [ 1219.778866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1219.810620] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1219.818521] LNet: Skipped 2 previous similar messages [ 1220.045674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1220.076513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1220.099983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1220.121603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1220.146373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1220.172202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1220.193790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1220.223711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1220.247291] LNet: 61326:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1220.268489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1220.298603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1220.321671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1220.353579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1220.392933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.421856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1220.475691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1220.532414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.973983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e remote_timeout -r 1 [ 1221.011408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e remote_timeout -r 1 [ 1221.054681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e remote_timeout -r 1 [ 1221.103598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e remote_timeout -r 1 [ 1221.153184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e remote_timeout -r 1 [ 1221.193736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e remote_timeout -r 1 [ 1221.235672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e remote_timeout -r 1 [ 1221.307763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e remote_timeout -r 1 [ 1221.366632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.117@tcp [ 1221.389931] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1221.394448] LNet: Skipped 1 previous similar message [ 1221.676330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.711928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1221.737524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.761300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.792825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.834348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.896134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.944264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.960778] LNet: 61449:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.979525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.036537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.090303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.152988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.192217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.251489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1222.299730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1222.331982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.908086] LNet: 61526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.914939] LNetError: 61526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.919906] LNet: Removed LNI 192.168.202.17@tcp [ 1226.946298] LNet: Removed LNI 192.168.202.17@tcp1 [ 1227.384210] Key type .llcrypt unregistered [ 1227.392867] Key type ._llcrypt unregistered [ 1246.735824] Key type ._llcrypt registered [ 1246.743284] Key type .llcrypt registered [ 1248.344637] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:31:13 (1768555873) [ 1248.981533] Key type .llcrypt unregistered [ 1248.984366] Key type ._llcrypt unregistered [ 1255.617668] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1255.816087] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1255.819395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1255.829051] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1258.454462] Key type ._llcrypt registered [ 1258.458866] Key type .llcrypt registered [ 1258.672548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.686750] alg: No test for adler32 (adler32-zlib) [ 1259.656570] LNet: 62599:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1259.666494] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1259.709372] LNet: 62613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.722399] LNetError: 62613:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.737818] LNet: Removed LNI 192.168.202.17@tcp [ 1260.356578] Key type .llcrypt unregistered [ 1260.359713] Key type ._llcrypt unregistered [ 1265.516599] Key type ._llcrypt registered [ 1265.519587] Key type .llcrypt registered [ 1265.655133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1265.664882] alg: No test for adler32 (adler32-zlib) [ 1266.589423] LNet: 62868:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1266.607495] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1266.609765] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1266.654703] LNet: 62882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.659724] LNetError: 62882:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.663963] LNet: Removed LNI 192.168.202.17@tcp [ 1266.681630] LNet: Removed LNI 10.1.2.3@tcp1 [ 1267.504351] Key type .llcrypt unregistered [ 1267.506177] Key type ._llcrypt unregistered [ 1273.296797] Key type ._llcrypt registered [ 1273.300662] Key type .llcrypt registered [ 1273.549109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1273.574694] alg: No test for adler32 (adler32-zlib) [ 1274.509199] LNet: 63137:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1274.518923] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1274.520790] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1274.552513] LNet: 63151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.556829] LNetError: 63151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.560646] LNet: Removed LNI 192.168.202.17@tcp [ 1274.571468] LNet: Removed LNI 10.1.2.3@tcp [ 1275.329557] Key type .llcrypt unregistered [ 1275.331088] Key type ._llcrypt unregistered [ 1282.602713] Key type ._llcrypt registered [ 1282.605383] Key type .llcrypt registered [ 1282.854932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1282.870375] alg: No test for adler32 (adler32-zlib) [ 1283.814394] LNet: 63407:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1283.825190] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1283.828602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1283.867253] LNet: 63421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.874461] LNetError: 63421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.881117] LNet: Removed LNI 192.168.202.17@tcp [ 1283.893031] LNet: Removed LNI 10.1.2.3@tcp [ 1284.455416] Key type .llcrypt unregistered [ 1284.457271] Key type ._llcrypt unregistered [ 1290.284712] Key type ._llcrypt registered [ 1290.289822] Key type .llcrypt registered [ 1290.513507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1290.523619] alg: No test for adler32 (adler32-zlib) [ 1291.450655] LNet: 63681:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1291.467599] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1291.471035] LNetError: 63681:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1291.478487] LNetError: Error -100 starting up LNI tcp [ 1291.501430] LNet: Removed LNI 192.168.202.17@tcp [ 1292.138625] Key type .llcrypt unregistered [ 1292.140711] Key type ._llcrypt unregistered [ 1303.185955] Key type ._llcrypt registered [ 1303.191455] Key type .llcrypt registered [ 1304.463610] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:32:10 (1768555930) [ 1305.183128] Key type .llcrypt unregistered [ 1305.185142] Key type ._llcrypt unregistered [ 1311.915975] Key type ._llcrypt registered [ 1311.921665] Key type .llcrypt registered [ 1312.184417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1312.197586] alg: No test for adler32 (adler32-zlib) [ 1313.127416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1313.157970] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1320.293396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1321.089005] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e network_timeout -r 1 [ 1321.143784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e network_timeout -r 1 [ 1321.219234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1321.239873] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1321.383465] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1321.504537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1321.548638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1321.578767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1321.612599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1321.641700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1321.679849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1321.707771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1321.733890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1321.745142] LNet: 64930:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1321.769102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1321.814371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1321.864641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1321.905431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1321.933195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1321.959867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1322.261234] LNet: 64968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.267569] LNetError: 64968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.280974] LNet: Removed LNI 192.168.202.17@tcp [ 1322.558658] Key type .llcrypt unregistered [ 1322.561626] Key type ._llcrypt unregistered [ 1340.233985] Key type ._llcrypt registered [ 1340.239447] Key type .llcrypt registered [ 1340.494525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1340.510275] alg: No test for adler32 (adler32-zlib) [ 1341.514822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1341.550701] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1349.514826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1359.319580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1359.371329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1359.397614] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1361.202240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e network_timeout -r 1 [ 1361.234932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e network_timeout -r 1 [ 1361.274387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e network_timeout -r 1 [ 1361.308329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e network_timeout -r 1 [ 1361.343311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e network_timeout -r 1 [ 1361.380627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e network_timeout -r 1 [ 1361.419333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e network_timeout -r 1 [ 1361.465357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e network_timeout -r 1 [ 1361.551621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1361.576134] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1361.897578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.947305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1361.980981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1362.017107] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1362.031711] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.117@tcp [ 1362.045848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1362.088200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1362.141520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1362.194750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.251833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.280732] LNet: 65840:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1362.307554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.339620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.370422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.401194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.436417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.472657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1367.497622] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1367.506683] LNet: Skipped 2 previous similar messages [ 1367.951433] LNet: 65913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.961858] LNetError: 65913:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.991486] LNet: Removed LNI 192.168.202.17@tcp [ 1368.996215] LNetError: 65913:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.026953] LNet: Removed LNI 192.168.202.17@tcp1 [ 1369.406721] Key type .llcrypt unregistered [ 1369.409226] Key type ._llcrypt unregistered [ 1386.518080] Key type ._llcrypt registered [ 1386.527953] Key type .llcrypt registered [ 1388.198914] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:33:33 (1768556013) [ 1388.742155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1388.759760] alg: No test for adler32 (adler32-zlib) [ 1389.897699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.006237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1390.063274] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1390.937401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1390.954661] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1391.986772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 1392.134831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1392.196274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1392.242517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1392.293464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1392.345802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1392.387120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.17@tcp [ 1392.410502] LNetError: 67012:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.17@tcp to recovery [ 1393.055298] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1393.055726] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1395.103645] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1396.064844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1399.199454] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1399.514669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1399.545959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1399.605355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.630520] LNet: 67119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.637326] LNetError: 67119:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.646757] LNet: Removed LNI 192.168.202.17@tcp [ 1399.659299] LNet: Removed LNI 192.168.202.17@tcp1 [ 1399.683980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.736503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.761514] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1400.387587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1400.403642] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1401.172368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 1401.279649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.386386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1401.430711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1401.468963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1401.501762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1401.536184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1401.581450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.17@tcp [ 1401.606920] LNetError: 67392:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.17@tcp to recovery [ 1401.763057] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1404.056469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1407.903400] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1407.911270] LNet: Skipped 1 previous similar message [ 1413.169444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1413.193162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1413.226684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1419.619226] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:34:05 (1768556045) [ 1419.955613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.988299] LNet: 68110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.007919] LNetError: 68110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.019409] LNetError: 68110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1420.030888] LNet: Removed LNI 192.168.202.17@tcp [ 1420.053684] LNet: Removed LNI 192.168.202.17@tcp1 [ 1420.073336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.165777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1420.194788] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1420.921870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1420.936640] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1421.587985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 1421.648356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1421.676041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1421.704199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1421.735320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1421.778258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1421.822142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.17@tcp --health 0 [ 1421.923813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1422.175803] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.17@tcp [ 1422.183804] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1422.194499] LNet: Skipped 1 previous similar message [ 1437.560383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1437.710899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1437.748655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.806192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.17@tcp --health 500 [ 1437.921943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1444.505044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1444.576344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.594306] LNet: 68601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.598857] LNetError: 68601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.603888] LNetError: 68601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1444.609656] LNet: Removed LNI 192.168.202.17@tcp [ 1444.621328] LNet: Removed LNI 192.168.202.17@tcp1 [ 1444.634786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.683876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.703732] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1445.223135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1445.239508] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1445.746941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 1445.773849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1445.823240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1445.847532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1445.872195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1445.896577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1445.922543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1445.945545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.17@tcp --health 0 [ 1446.687758] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1446.697046] LNet: Skipped 3 previous similar messages [ 1447.187247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1457.528376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1457.574779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1457.611505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1464.261992] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:34:50 (1768556090) [ 1464.853108] LNet: 69647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.861422] LNetError: 69647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.872198] LNetError: 69647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1464.880263] LNet: Removed LNI 192.168.202.17@tcp [ 1464.897775] LNet: Removed LNI 192.168.202.17@tcp1 [ 1465.494876] Key type .llcrypt unregistered [ 1465.497892] Key type ._llcrypt unregistered [ 1471.906707] Key type ._llcrypt registered [ 1471.908107] Key type .llcrypt registered [ 1472.142412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1472.161498] alg: No test for adler32 (adler32-zlib) [ 1473.127027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1473.163655] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1482.842640] Lustre: DEBUG MARKER: Initial discovery [ 1483.090741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.117@tcp [ 1485.458951] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1485.623156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1485.649638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.117@tcp [ 1485.663993] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1485.680533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1485.709178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.117@tcp -r 1 -m GET -l 3 [ 1485.728612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.117@tcp -r 1 -m GET -e local_error [ 1485.752940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.117@tcp -r 1 -m PUT -l 6 [ 1486.495299] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1486.608840] Lustre: DEBUG MARKER: Force oleg217-client.virtnet to discover 192.168.202.117@tcp (in background) [ 1486.800366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.117@tcp [ 1488.015752] Lustre: DEBUG MARKER: Force oleg217-server to discover 192.168.202.17@tcp [ 1489.887400] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1490.517907] Lustre: DEBUG MARKER: Wait for 70406 [ 1491.919859] Lustre: DEBUG MARKER: Finished wait on 70406 [ 1492.122181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1492.170935] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1492.172682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1492.414547] LNet: 70853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.419548] LNetError: 70853:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.437163] LNet: Removed LNI 192.168.202.17@tcp [ 1492.852814] Key type .llcrypt unregistered [ 1492.864371] Key type ._llcrypt unregistered [ 1511.564654] Key type ._llcrypt registered [ 1511.567025] Key type .llcrypt registered [ 1512.700633] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:35:38 (1768556138) [ 1513.128283] Key type .llcrypt unregistered [ 1513.130044] Key type ._llcrypt unregistered [ 1520.337456] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1520.649191] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1520.651922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1520.656045] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1522.219702] Key type ._llcrypt registered [ 1522.224086] Key type .llcrypt registered [ 1522.366700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.388089] alg: No test for adler32 (adler32-zlib) [ 1523.371124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.416866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1523.437473] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1524.093864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1524.122616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1533.902495] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:35:59 (1768556159) [ 1534.132607] LNet: 72908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.145389] LNetError: 72908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.165523] LNet: Removed LNI 192.168.202.17@tcp [ 1534.188837] LNet: Removed LNI 10.1.2.3@tcp [ 1534.862714] Key type .llcrypt unregistered [ 1534.866413] Key type ._llcrypt unregistered [ 1543.272674] Key type ._llcrypt registered [ 1543.275334] Key type .llcrypt registered [ 1543.580778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1543.628826] alg: No test for adler32 (adler32-zlib) [ 1544.685733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.764918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1544.790452] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1545.599832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1545.630302] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1547.133894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.17@tcp [ 1548.191169] LNetError: Unexpected error -22 connecting to 192.168.202.17@tcp at host 192.168.202.17:988 [ 1555.579216] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:36:21 (1768556181) [ 1556.087059] LNet: 74044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.111754] LNetError: 74044:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.127818] LNet: Removed LNI 192.168.202.17@tcp [ 1556.146198] LNet: Removed LNI 10.1.2.3@tcp [ 1557.665834] Key type .llcrypt unregistered [ 1557.673219] Key type ._llcrypt unregistered [ 1563.288935] Key type ._llcrypt registered [ 1563.292926] Key type .llcrypt registered [ 1563.596278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1563.630191] alg: No test for adler32 (adler32-zlib) [ 1564.543403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.597847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1564.629966] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1565.629894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1565.658079] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 1566.518887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.17@tcp1 --nid 192.168.202.17@tcp2 [ 1570.619208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp1 192.168.202.17@tcp2 [ 1570.645064] LNetError: 74875:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.17@tcp1 to 192.168.202.17@tcp2 - Route table may be misconfigured [ 1570.678504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp2 192.168.202.17@tcp1 [ 1578.448424] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:36:44 (1768556204) [ 1578.754623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.818179] LNet: 75448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.838972] LNetError: 75448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.871193] LNet: Removed LNI 192.168.202.17@tcp1 [ 1579.877729] LNetError: 75448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.917510] LNet: Removed LNI 192.168.202.17@tcp2 [ 1579.952637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.001711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.023384] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1580.772696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1580.790806] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1581.603492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 1581.631925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp 192.168.202.17@tcp [ 1581.663811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp1 192.168.202.17@tcp1 [ 1581.706601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.17@tcp -d 192.168.202.17@tcp -e network_timeout [ 1581.740763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -e network_timeout [ 1581.782307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -e network_timeout [ 1581.849343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -e network_timeout [ 1581.878553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1581.900114] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1581.914552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1582.003276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1582.051413] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1582.122489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1589.236682] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:36:54 (1768556214) [ 1589.460759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.499900] LNet: 76310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.510757] LNetError: 76310:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.519620] LNet: Removed LNI 192.168.202.17@tcp [ 1590.571188] LNet: Removed LNI 192.168.202.17@tcp1 [ 1590.605048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.733533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1590.997852] LNet: 76347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.459834] Key type .llcrypt unregistered [ 1591.463648] Key type ._llcrypt unregistered [ 1604.252232] Key type ._llcrypt registered [ 1604.254202] Key type .llcrypt registered [ 1605.239729] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:37:11 (1768556231) [ 1605.974861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1605.988914] alg: No test for adler32 (adler32-zlib) [ 1607.050438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.928874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1610.606299] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:37:16 (1768556236) [ 1610.941881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.961143] LNet: 77442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.981749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.043897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.059994] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1611.544460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1611.566994] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1612.234877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1612.272572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp1 [ 1612.304383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp1 [ 1618.880557] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:37:24 (1768556244) [ 1621.247354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg217-server [ 1622.919110] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:37:28 (1768556248) [ 1625.174315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg217-server [ 1627.027670] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:37:33 (1768556253) [ 1629.203783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg217-server [ 1631.315428] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:37:37 (1768556257) [ 1633.090607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg217-server [ 1634.837453] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:37:40 (1768556260) [ 1637.316695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg217-server [ 1639.451246] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:37:44 (1768556264) [ 1641.783373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg217-server [ 1643.614949] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:37:49 (1768556269) [ 1646.227947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg217-server [ 1648.579941] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:37:54 (1768556274) [ 1650.532620] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg217-server [ 1652.227311] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:37:58 (1768556278) [ 1653.897974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg217-server [ 1655.887813] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:38:01 (1768556281) [ 1656.100302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.118322] LNet: 81319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.123556] LNetError: 81319:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.127992] LNet: Removed LNI 192.168.202.17@tcp [ 1657.201501] LNet: Removed LNI 192.168.202.17@tcp1 [ 1657.245512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.320593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1657.339421] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1657.890619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1657.972478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1658.173985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.202841] LNet: 81467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.209793] LNetError: 81467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.215063] LNetError: 81467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1659.237820] LNet: Removed LNI 192.168.202.17@tcp [ 1659.265840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.315572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1659.342648] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1660.083737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1660.241325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.258952] LNet: 81612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.264933] LNetError: 81612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.287883] LNet: Removed LNI 192.168.202.17@tcp [ 1661.339406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.430842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.476255] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1662.218185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1662.308605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1662.495221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.513913] LNetError: 81759:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.529632] LNet: Removed LNI 192.168.202.17@tcp [ 1662.565253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.657740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1662.718482] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1663.467670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1663.717591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.805844] LNet: Removed LNI 192.168.202.17@tcp [ 1664.856416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.916651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.964552] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1665.892589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1666.020928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1666.321700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.353669] LNet: 82052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.356912] LNet: 82052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1667.430089] LNet: Removed LNI 192.168.202.17@tcp [ 1667.488767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.596447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1667.649381] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1668.838769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1669.165450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.211988] LNetError: 82197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1669.221554] LNetError: 82197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1669.251747] LNet: Removed LNI 192.168.202.17@tcp [ 1669.291557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.389883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.427352] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1670.378259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1670.506869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1670.872026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.985064] LNet: Removed LNI 192.168.202.17@tcp [ 1672.052544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.142095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1672.168099] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1672.734379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1672.990035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.094192] LNet: Removed LNI 192.168.202.17@tcp [ 1674.159415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.258172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.310611] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1675.272356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1675.414444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1675.648495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.675632] LNet: 82637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.683671] LNet: 82637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1676.711225] LNet: Removed LNI 192.168.202.17@tcp [ 1676.759863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.901094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1676.947735] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1677.883652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1678.227904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.273049] LNetError: 82783:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.278039] LNetError: 82783:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1678.299770] LNet: Removed LNI 192.168.202.17@tcp [ 1678.323904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.419528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.447515] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1679.108949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1679.168545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1679.407026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.485806] LNet: Removed LNI 192.168.202.17@tcp [ 1680.513602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.568696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1680.618090] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1681.258037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1681.591575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.664342] LNet: Removed LNI 192.168.202.17@tcp [ 1682.703497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.761453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.807886] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1683.526590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1683.602466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.639909] LNet: Removed LNI 192.168.202.17@tcp [ 1683.661684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.743710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.776419] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1684.813808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1691.932754] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:38:37 (1768556317) [ 1692.236318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.281714] LNet: 83912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.285683] LNet: 83912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1692.310751] LNet: Removed LNI 192.168.202.17@tcp [ 1692.321942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.363306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1692.381629] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/180] [ 1693.678550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.709277] LNet: Removed LNI 192.168.202.17@tcp231 [ 1693.733608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.788709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1693.814542] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/180] [ 1693.821570] LNet: 84058:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1694.120661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1694.137590] LNet: Removed LNI 192.168.202.17@tcp231 [ 1694.150349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1694.168026] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/180] [ 1695.136879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.157417] LNetError: 84202:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.162493] LNetError: 84202:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1695.178807] LNet: Removed LNI 192.168.202.17@tcp231 [ 1695.204695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.275903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1695.315643] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/0] [ 1695.320865] LNet: 84210:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1695.757851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1695.814792] LNet: Removed LNI 192.168.202.17@tcp231 [ 1695.848241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1695.891562] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/0] [ 1697.475415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.527646] LNet: Removed LNI 192.168.202.17@tcp231 [ 1697.557879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.665989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1697.688553] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/60] [ 1697.694242] LNet: 84362:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1698.068660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1698.094415] LNet: Removed LNI 192.168.202.17@tcp231 [ 1698.110596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1698.133190] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/60] [ 1698.965849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.988768] LNet: Removed LNI 192.168.202.17@tcp231 [ 1699.008455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.055888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1699.080112] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/180] [ 1699.412486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1699.467625] LNet: Removed LNI 192.168.202.17@tcp231 [ 1699.494333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1699.520609] LNet: Added LNI 192.168.202.17@tcp231 [8/256/0/180] [ 1707.021505] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:38:52 (1768556332) [ 1707.251619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.320149] LNet: Removed LNI 192.168.202.17@tcp231 [ 1707.336826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.451644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1707.481765] LNet: Added LNI 192.168.202.17@tcp232 [8/256/0/180] [ 1708.240762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1708.283838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1708.313175] LNet: Removed LNI 192.168.202.17@tcp232 [ 1708.335512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1708.368515] LNet: Added LNI 192.168.202.17@tcp232 [8/256/0/180] [ 1709.391312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1709.430222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1709.465663] LNet: Removed LNI 192.168.202.17@tcp232 [ 1709.481162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1709.514143] LNet: Added LNI 192.168.202.17@tcp232 [8/256/0/180] [ 1710.150468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1710.169036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1710.188886] LNet: Removed LNI 192.168.202.17@tcp232 [ 1710.195644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1710.207393] LNet: Added LNI 192.168.202.17@tcp232 [8/256/0/180] [ 1710.706726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1710.731512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1710.759996] LNet: Removed LNI 192.168.202.17@tcp232 [ 1710.772405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1716.902963] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:39:02 (1768556342) [ 1718.104270] Key type .llcrypt unregistered [ 1718.116030] Key type ._llcrypt unregistered [ 1724.344329] Key type ._llcrypt registered [ 1724.351389] Key type .llcrypt registered [ 1724.602361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1724.628303] alg: No test for adler32 (adler32-zlib) [ 1725.639489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1725.690672] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1733.685207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1744.591440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.644232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1744.685837] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1746.856907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_interrupt -r 2 [ 1746.900335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_interrupt -r 2 [ 1746.933532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_interrupt -r 2 [ 1746.976568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_interrupt -r 2 [ 1747.023125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_interrupt -r 2 [ 1747.056885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_interrupt -r 2 [ 1747.097343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_interrupt -r 2 [ 1747.132945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_interrupt -r 2 [ 1747.189228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp 192.168.202.117@tcp [ 1747.226263] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1747.234164] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1747.608673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1747.652441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1747.707930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1747.745735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1747.791214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1747.832342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1747.879503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1747.937325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1747.981510] LNet: 87013:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1748.014114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1748.071510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1748.128993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1748.201992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1748.241504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1748.292429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1749.111452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_dropped -r 2 [ 1749.198188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_dropped -r 2 [ 1749.236729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_dropped -r 2 [ 1749.268440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_dropped -r 2 [ 1749.305195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_dropped -r 2 [ 1749.339959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_dropped -r 2 [ 1749.366397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_dropped -r 2 [ 1749.392533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_dropped -r 2 [ 1749.425852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp 192.168.202.117@tcp [ 1749.445511] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1749.662944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1749.692278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1749.718278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1749.739661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1749.764918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1749.791251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1749.817225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1749.839962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1749.853509] LNet: 87133:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1749.871948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1749.902622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1749.941994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1749.979487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1750.019554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1750.056822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1750.705871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_aborted -r 2 [ 1750.740081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_aborted -r 2 [ 1750.766283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_aborted -r 2 [ 1750.799921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_aborted -r 2 [ 1750.834490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_aborted -r 2 [ 1750.882910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_aborted -r 2 [ 1750.944638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_aborted -r 2 [ 1750.986357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_aborted -r 2 [ 1751.028771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp 192.168.202.117@tcp [ 1751.056267] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1751.361269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1751.389961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1751.425824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1751.453880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1751.488919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1751.521068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1751.563434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1751.589962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1751.607656] LNet: 87254:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1751.623685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1751.658653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1751.683464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1751.711396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1751.735419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1751.761607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1752.238521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_no_route -r 2 [ 1752.266777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_no_route -r 2 [ 1752.304703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_no_route -r 2 [ 1752.342633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_no_route -r 2 [ 1752.380655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_no_route -r 2 [ 1752.419269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_no_route -r 2 [ 1752.450221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_no_route -r 2 [ 1752.479484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_no_route -r 2 [ 1752.513035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp 192.168.202.117@tcp [ 1752.780858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1752.810819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1752.842086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1752.869762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1752.890697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1752.914624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1752.936672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1752.973956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1753.011354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1753.048367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1753.105142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1753.154749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1753.209700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1753.252961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1753.785378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e local_timeout -r 2 [ 1753.814383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e local_timeout -r 2 [ 1753.843163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e local_timeout -r 2 [ 1753.869636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e local_timeout -r 2 [ 1753.902965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e local_timeout -r 2 [ 1753.930601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e local_timeout -r 2 [ 1753.966819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e local_timeout -r 2 [ 1753.995381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e local_timeout -r 2 [ 1754.028837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp 192.168.202.117@tcp [ 1754.048780] LNet: There was an unexpected network error while writing to 192.168.202.117: rc = -22 [ 1754.053220] LNet: Skipped 1 previous similar message [ 1754.288247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1754.320699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1754.342786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1754.362976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1754.389962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1754.435349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1754.501356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1754.540942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1754.569086] LNet: 87494:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1754.582168] LNet: 87494:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1754.604530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1754.651755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1754.680795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1754.723417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1754.771401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1754.816867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1755.341726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp -m GET -e remote_dropped -r 2 [ 1755.389657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.117@tcp1 -m GET -e remote_dropped -r 2 [ 1755.425517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp -m GET -e remote_dropped -r 2 [ 1755.498317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp -d 192.168.202.17@tcp1 -m GET -e remote_dropped -r 2 [ 1755.541637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp -m GET -e remote_dropped -r 2 [ 1755.580688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.117@tcp1 -m GET -e remote_dropped -r 2 [ 1755.619420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp -m GET -e remote_dropped -r 2 [ 1755.667182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.17@tcp1 -d 192.168.202.17@tcp1 -m GET -e remote_dropped -r 2 [ 1755.701835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.17@tcp 192.168.202.117@tcp [ 1755.721543] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.117@tcp [ 1755.931162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1755.964232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1755.989208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1756.017106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1756.057623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1756.083552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1756.113707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1756.151702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1756.208828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1756.261872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1756.289783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1756.313122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1756.339681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1756.373758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1762.856133] LNet: 87687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.862063] LNetError: 87687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.871871] LNet: Removed LNI 192.168.202.17@tcp [ 1763.880748] LNetError: 87687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.906605] LNet: Removed LNI 192.168.202.17@tcp1 [ 1764.569691] Key type .llcrypt unregistered [ 1764.578408] Key type ._llcrypt unregistered [ 1781.832271] Key type ._llcrypt registered [ 1781.835860] Key type .llcrypt registered [ 1783.668490] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:40:08 (1768556408) [ 1784.340334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.375530] alg: No test for adler32 (adler32-zlib) [ 1785.369883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.453037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1785.487590] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1786.472624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1786.512371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.17@tcp [ 1786.537371] LNetError: 88665:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.17@tcp to recovery [ 1786.666508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.17@tcp [ 1786.801452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1794.059509] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:40:19 (1768556419) [ 1794.517742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.550641] LNet: 89255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.557420] LNetError: 89255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.588849] LNet: Removed LNI 192.168.202.17@tcp [ 1794.618704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.698638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.751882] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1795.744623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1795.775685] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 1796.955625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.17@tcp [ 1796.970955] LNetError: 89521:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.202.17@tcp to recovery [ 1797.727396] LNet: 1 local NIs in recovery (showing 1): 192.168.202.17@tcp [ 1797.729234] LNet: There was an unexpected network error while writing to 192.168.202.17: rc = -22 [ 1798.302909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.17@tcp [ 1809.217284] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:40:35 (1768556435) [ 1810.980165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg217-server [ 1813.031792] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:40:38 (1768556438) [ 1813.307655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.332968] LNet: 90477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.341759] LNetError: 90477:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.351211] LNet: Removed LNI 192.168.202.17@tcp [ 1813.375866] LNet: Removed LNI 192.168.202.17@tcp1 [ 1813.433352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.503268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.517375] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1814.194216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1814.249699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1814.288915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1814.360647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1814.407183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1814.441793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1820.904388] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:40:47 (1768556447) [ 1821.291721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.334687] LNet: 91204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.340387] LNetError: 91204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.347396] LNetError: 91204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1821.367358] LNet: Removed LNI 192.168.202.17@tcp [ 1821.386857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.456823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.486477] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1829.349596] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:40:54 (1768556454) [ 1829.420874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.459809] LNet: 91900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.472736] LNetError: 91900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.495179] LNet: Removed LNI 192.168.202.17@tcp [ 1829.540083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.639264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1829.671586] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1837.506951] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:41:03 (1768556463) [ 1838.853906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1840.590659] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:41:06 (1768556466) [ 1840.970211] LNet: 92942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.980522] LNetError: 92942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.998561] LNet: Removed LNI 192.168.202.17@tcp [ 1841.489692] Key type .llcrypt unregistered [ 1841.491987] Key type ._llcrypt unregistered [ 1847.080948] Key type ._llcrypt registered [ 1847.084126] Key type .llcrypt registered [ 1847.294478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1847.342947] alg: No test for adler32 (adler32-zlib) [ 1848.334767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1848.382768] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1855.827416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 1862.827584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.117@tcp [ 1863.291503] LNet: 93355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.305848] LNetError: 93355:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.321268] LNet: Removed LNI 192.168.202.17@tcp [ 1863.798239] Key type .llcrypt unregistered [ 1863.803114] Key type ._llcrypt unregistered [ 1875.866997] Key type ._llcrypt registered [ 1875.870356] Key type .llcrypt registered [ 1877.331606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1878.908898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1880.927276] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:41:46 (1768556506) [ 1881.326804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.339240] alg: No test for adler32 (adler32-zlib) [ 1882.325112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.445209] LNet: 94439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.148831] Key type .llcrypt unregistered [ 1883.153761] Key type ._llcrypt unregistered [ 1888.811399] Key type ._llcrypt registered [ 1888.814648] Key type .llcrypt registered [ 1888.998222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1889.012875] alg: No test for adler32 (adler32-zlib) [ 1889.949615] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1889.955652] LNet: 94692:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1889.982586] LNet: 94692:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1890.041290] LNetError: 94704:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.21@tcp is being used as a gateway but routing feature is not turned on [ 1890.053380] LNetError: 94704:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.202.21@tcp has gone from up to down [ 1890.491590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1890.510645] LNet: 94750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.516298] LNetError: 94750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.538647] LNet: Removed LNI 192.168.202.17@tcp [ 1890.554587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.597158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1890.615084] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1890.619627] LNet: 94757:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1890.625309] LNet: 94757:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1890.695305] LNetError: 94757:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.18@tcp is a gateway. Can not delete it [ 1890.702310] LNetError: 94757:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.202.18@tcp is a router. Can not be deleted [ 1890.708761] LNet: 94757:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1890.738892] LNetError: 94754:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.21@tcp is being used as a gateway but routing feature is not turned on [ 1890.749535] LNetError: 94754:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 1890.758158] LNetError: 94754:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.202.21@tcp has gone from up to down [ 1890.776240] LNetError: 94754:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1897.145831] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:42:02 (1768556522) [ 1898.789735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg217-server [ 1900.543392] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:42:06 (1768556526) [ 1902.693352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg217-server [ 1904.540749] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:42:10 (1768556530) [ 1906.480796] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 1908.599604] Key type lgssc registered [ 1909.878559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.551718] Key type lgssc unregistered [ 1929.000217] LNet: 97187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.008567] LNetError: 97187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.031438] LNet: Removed LNI 192.168.202.17@tcp [ 1929.654443] Key type .llcrypt unregistered [ 1929.660849] Key type ._llcrypt unregistered [ 1946.304553] Key type ._llcrypt registered [ 1946.307517] Key type .llcrypt registered [ 1947.736729] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:42:53 (1768556573) [ 1948.225263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.239455] alg: No test for adler32 (adler32-zlib) [ 1949.198260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.264157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1949.303919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1949.339275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1949.368546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1956.279760] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:43:01 (1768556581) [ 1956.459828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1956.514915] LNet: 98585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.572418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.685589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1956.730151] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1957.897412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.921230] LNet: 98736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.926710] LNetError: 98736:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.939249] LNet: Removed LNI 192.168.202.17@tcp [ 1957.969971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.032476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1958.066979] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1958.567901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.586282] LNetError: 98765:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.594444] LNet: Removed LNI 192.168.202.17@tcp [ 1958.610856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.650970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1958.681152] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1966.407659] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:43:11 (1768556591) [ 1966.540909] LNet: 99347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.544722] LNet: 99347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1966.559903] LNetError: 99347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.581744] LNet: Removed LNI 192.168.202.17@tcp [ 1967.477353] Key type .llcrypt unregistered [ 1967.482678] Key type ._llcrypt unregistered [ 1976.109254] Key type ._llcrypt registered [ 1976.115314] Key type .llcrypt registered [ 1976.547321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1976.586793] alg: No test for adler32 (adler32-zlib) [ 1977.676480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.856211] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 1977.866204] LNet: *** cfs_fail_loc=e003, val=100*** [ 1978.967566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 1978.996449] LNet: *** cfs_fail_loc=e003, val=100*** [ 1979.000950] LNet: Skipped 2 previous similar messages [ 1979.019922] Ping reply received with mapping: [ 1979.027065] found metadata reply NI[192.168.202.17@tcp] [ 1979.035378] 11 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1979.042116] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1979.058512] 5a 5a 5a 5a 5a 5a 5a 5a [ 1979.236439] LNet: 99751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.258960] LNetError: 99751:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.331567] LNet: Removed LNI 192.168.202.17@tcp [ 1981.160604] Key type .llcrypt unregistered [ 1981.171552] Key type ._llcrypt unregistered [ 1993.852883] Key type ._llcrypt registered [ 1993.854785] Key type .llcrypt registered [ 1995.392858] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:43:41 (1768556621) [ 1995.858291] Key type .llcrypt unregistered [ 1995.859422] Key type ._llcrypt unregistered [ 2002.689182] Key type ._llcrypt registered [ 2002.690668] Key type .llcrypt registered [ 2003.165456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2003.194520] alg: No test for adler32 (adler32-zlib) [ 2004.564208] LNetError: 100786:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2005.029608] Key type .llcrypt unregistered [ 2005.038292] Key type ._llcrypt unregistered [ 2011.663132] Key type ._llcrypt registered [ 2011.665147] Key type .llcrypt registered [ 2013.022095] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:43:58 (1768556638) [ 2013.590699] Key type .llcrypt unregistered [ 2013.596476] Key type ._llcrypt unregistered [ 2020.056884] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2020.061958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2022.559728] Key type ._llcrypt registered [ 2022.562404] Key type .llcrypt registered [ 2022.778510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2022.811882] alg: No test for adler32 (adler32-zlib) [ 2023.699217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.753917] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2032.424508] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:44:18 (1768556658) [ 2032.600785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.646611] LNet: 102559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.660134] LNetError: 102559:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2032.680843] LNet: Removed LNI 10.1.2.3@tcp [ 2032.707779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.222908] Key type .llcrypt unregistered [ 2033.225430] Key type ._llcrypt unregistered [ 2039.929415] Key type ._llcrypt registered [ 2039.934162] Key type .llcrypt registered [ 2040.120864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2040.134555] alg: No test for adler32 (adler32-zlib) [ 2041.085634] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2041.174557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 2041.355883] LNet: 102844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.360908] LNetError: 102844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.406410] LNet: Removed LNI 192.168.202.17@tcp [ 2043.180872] Key type .llcrypt unregistered [ 2043.184875] Key type ._llcrypt unregistered [ 2055.550106] Key type ._llcrypt registered [ 2055.561463] Key type .llcrypt registered [ 2057.039091] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:44:42 (1768556682) [ 2057.605849] Key type .llcrypt unregistered [ 2057.607694] Key type ._llcrypt unregistered [ 2062.839594] Key type ._llcrypt registered [ 2062.842831] Key type .llcrypt registered [ 2063.092423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2063.114324] alg: No test for adler32 (adler32-zlib) [ 2070.503306] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:44:56 (1768556696) [ 2070.650990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.713189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.798054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2070.844766] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2071.483921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2071.517186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2071.552795] LNet: Removed LNI 192.168.202.17@tcp [ 2071.764122] LNet: 104580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.115757] Key type .llcrypt unregistered [ 2072.118572] Key type ._llcrypt unregistered [ 2083.354970] Key type ._llcrypt registered [ 2083.357661] Key type .llcrypt registered [ 2084.497440] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:45:10 (1768556710) [ 2085.170223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2085.188888] alg: No test for adler32 (adler32-zlib) [ 2086.157546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.248030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2086.298597] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2087.092224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 2087.179538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.17@tcp [ 2087.242319] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2087.250823] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2087.255462] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2087, 0, 0 [ 2087.260557] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2087.263388] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2087.271260] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2087.278318] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2087, 0, 0 [ 2087.284209] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2087.291209] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2087.298192] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2087.307530] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2087, 0, 2137 [ 2087.317429] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2087.327942] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2087.332178] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2087.336719] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2087, 0, 2137 [ 2087.341586] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2094.300532] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:45:20 (1768556720) [ 2094.738438] LNet: 106118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.745148] LNetError: 106118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.761951] LNet: Removed LNI 192.168.202.17@tcp [ 2095.379685] Key type .llcrypt unregistered [ 2095.387243] Key type ._llcrypt unregistered [ 2101.401715] Key type ._llcrypt registered [ 2101.404397] Key type .llcrypt registered [ 2101.576344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2101.592827] alg: No test for adler32 (adler32-zlib) [ 2102.563939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2102.587072] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2110.183341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 2119.867771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.980707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2120.011695] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 2121.323688] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2121.432993] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2121.435636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2121.438025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2123.960132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2123.982128] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2135.570458] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:46:01 (1768556761) [ 2135.958071] LNet: 107636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.966465] LNetError: 107636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.971500] LNet: Removed LNI 192.168.202.17@tcp [ 2136.991277] LNet: Removed LNI 192.168.202.17@tcp1 [ 2136.993446] LNetError: 107636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2137.000432] LNetError: 107636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2137.015676] LNet: Removed LNI 10.1.2.3@tcp99 [ 2137.807256] Key type .llcrypt unregistered [ 2137.811183] Key type ._llcrypt unregistered [ 2146.238641] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2146.249385] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2149.036358] Key type ._llcrypt registered [ 2149.038350] Key type .llcrypt registered [ 2149.373233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2149.408077] alg: No test for adler32 (adler32-zlib) [ 2150.479464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.585673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.636759] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2151.450631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2151.470827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2153.744917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2153.799120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2153.889429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2153.908697] LNetError: 108378:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2153.948835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2153.995508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2154.606962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2154.770615] LNet: 108398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.786190] LNetError: 108398:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.812206] LNet: Removed LNI 192.168.202.17@tcp [ 2154.835905] LNet: Removed LNI 10.1.2.3@tcp [ 2155.468461] Key type .llcrypt unregistered [ 2155.472987] Key type ._llcrypt unregistered [ 2170.529067] Key type ._llcrypt registered [ 2170.532133] Key type .llcrypt registered [ 2171.801606] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:46:37 (1768556797) [ 2172.280168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.292839] alg: No test for adler32 (adler32-zlib) [ 2173.264811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.329117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2173.344440] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2173.995830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.17@tcp [ 2174.054816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg217-client@tcp [ 2180.289326] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:46:46 (1768556806) [ 2180.451431] LNet: 109928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.458237] LNetError: 109928:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.497494] LNet: Removed LNI 192.168.202.17@tcp [ 2181.102728] Key type .llcrypt unregistered [ 2181.105964] Key type ._llcrypt unregistered [ 2187.662739] Key type ._llcrypt registered [ 2187.666735] Key type .llcrypt registered [ 2187.882419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2187.901882] alg: No test for adler32 (adler32-zlib) [ 2188.868903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2188.900845] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2188.947372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2195.820784] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:47:01 (1768556821) [ 2197.607299] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2200.135822] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:47:05 (1768556825) [ 2201.958863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2204.224649] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 04:47:09 (1768556829) [ 2204.382270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.413301] LNet: 111340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.423389] LNetError: 111340:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.442380] LNet: Removed LNI 192.168.202.17@tcp [ 2204.477499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.568811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2204.604129] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2205.453184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2205.539855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2212.224330] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:47:18 (1768556838) [ 2212.448814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.466196] LNet: 112056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.480851] LNetError: 112056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.501331] LNet: Removed LNI 192.168.202.17@tcp [ 2212.515646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.561197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2212.589388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2213.243380] Key type .llcrypt unregistered [ 2213.249121] Key type ._llcrypt unregistered [ 2225.522309] Key type ._llcrypt registered [ 2225.523642] Key type .llcrypt registered [ 2227.232052] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:47:32 (1768556852) [ 2227.688823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2227.702753] alg: No test for adler32 (adler32-zlib) [ 2228.627907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.695217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.723581] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2229.385346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2229.448738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2236.398297] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:47:42 (1768556862) [ 2236.605950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.672396] LNet: 113608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.678568] LNetError: 113608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.701202] LNet: Removed LNI 192.168.202.17@tcp [ 2236.722279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.779377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2236.817667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2242.775345] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 04:47:48 (1768556868) [ 2243.004300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.033788] LNet: 114188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.073763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.185466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2243.479656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2243.507464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2243.551962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2243.984713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2244.048635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2244.208310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.267171] LNet: 114221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.309712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.407266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2244.850230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2244.920937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2245.039909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2245.875598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2245.909679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2252.798271] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 04:47:58 (1768556878) [ 2252.910095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.945778] LNet: 114819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.985988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.076156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2253.101792] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2253.703265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2253.722040] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2254.292593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2254.338674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2254.534934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.571701] LNetError: 115101:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.577174] LNet: Removed LNI 192.168.202.17@tcp [ 2254.593736] LNet: Removed LNI 192.168.202.17@tcp2 [ 2254.615761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.683325] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2254.688406] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2254.695735] LNet: 115107:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2255.160458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2255.422564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.481345] LNet: Removed LNI 192.168.202.17@tcp [ 2256.483391] LNetError: 115145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.498744] LNetError: 115145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2256.532201] LNet: Removed LNI 192.168.202.17@tcp2 [ 2256.587435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.658737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.707537] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2257.427674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2257.447225] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2258.252664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2258.343793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2258.437649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2258.723684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.759589] LNet: 115429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.766543] LNet: 115429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2258.772139] LNetError: 115429:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.789170] LNet: Removed LNI 192.168.202.17@tcp [ 2259.816283] LNet: Removed LNI 192.168.202.17@tcp2 [ 2259.887889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.991605] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2260.004384] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2260.007702] LNet: 115435:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2260.612528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2260.894158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2260.940793] LNet: Removed LNI 192.168.202.17@tcp [ 2261.992757] LNet: Removed LNI 192.168.202.17@tcp2 [ 2262.023054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.099448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2262.119636] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2262.864136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2262.892126] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2263.901650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2263.945811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2264.108868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2264.126936] LNetError: 115752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.130691] LNetError: 115752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2265.183256] LNet: Removed LNI 192.168.202.17@tcp [ 2265.249086] LNet: Removed LNI 192.168.202.17@tcp2 [ 2265.344462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.440446] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2265.447569] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2265.460699] LNet: 115758:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2266.210114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2266.504857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.554673] LNet: Removed LNI 192.168.202.17@tcp [ 2267.587423] LNet: Removed LNI 192.168.202.17@tcp2 [ 2267.644986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.787704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.812056] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2268.696698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2268.714883] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2269.503950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2269.554746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2269.596938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2269.757660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2269.775704] LNet: 116080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.779811] LNet: 116080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2270.816378] LNet: Removed LNI 192.168.202.17@tcp [ 2271.851401] LNet: Removed LNI 192.168.202.17@tcp2 [ 2271.881352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.964648] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2271.968566] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2271.972857] LNet: 116086:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2272.618352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2272.737862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.765947] LNetError: 116124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.772176] LNetError: 116124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2272.779580] LNet: Removed LNI 192.168.202.17@tcp [ 2273.839228] LNet: Removed LNI 192.168.202.17@tcp2 [ 2273.890883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.965672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2273.990458] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2274.705214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2274.722563] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2275.517935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2275.558672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2275.644463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2275.935432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.959518] LNet: Removed LNI 192.168.202.17@tcp [ 2276.972355] LNet: Removed LNI 192.168.202.17@tcp2 [ 2277.004063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.082112] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2277.088903] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2277.097436] LNet: 116426:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2277.565368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2277.878738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.943312] LNet: Removed LNI 192.168.202.17@tcp [ 2278.957550] LNet: Removed LNI 192.168.202.17@tcp2 [ 2278.990464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.047518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.071315] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2279.866560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2279.896495] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2280.917106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2280.959867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2280.982698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2281.024818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2281.369915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.399261] LNet: Removed LNI 192.168.202.17@tcp [ 2283.429628] LNet: Removed LNI 192.168.202.17@tcp2 [ 2283.456396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.539779] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2283.544056] LNet: Added LNI 192.168.202.17@tcp2 [8/256/0/180] [ 2284.289419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2292.300676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2293.978931] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 04:48:39 (1768556919) [ 2294.210139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.222746] LNet: 117545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.226232] LNet: 117545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2294.231266] LNetError: 117545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.235776] LNetError: 117545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2294.240749] LNet: Removed LNI 192.168.202.17@tcp [ 2295.271943] LNet: Removed LNI 192.168.202.17@tcp2 [ 2295.312678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.629972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2295.660320] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2296.370774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2296.392086] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2297.631384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2297.661412] LNet: Removed LNI 192.168.202.17@tcp [ 2297.674613] LNet: Removed LNI 10.1.2.3@tcp [ 2297.742427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.862418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2297.895241] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2297.902206] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2298.369920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2298.399668] LNet: Removed LNI 192.168.202.17@tcp [ 2298.415687] LNet: Removed LNI 10.1.2.3@tcp [ 2307.075686] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:48:52 (1768556932) [ 2307.155203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.195251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.335350] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2307.397685] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2307.400278] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2307.403069] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2309.898580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2309.996489] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2311.189885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2311.246825] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2314.796740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.17@tcp [ 2314.856560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2314.965816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.17@tcp -m PUT -e local_timeout -r 1 [ 2315.037884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2315.105690] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2315.115439] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2315.355339] LNetError: 118487:0:(lib-move.c:4947:lnet_parse()) 192.168.202.17@tcp, src 192.168.202.17@tcp: Dropping PUT (error -108 looking up sender) [ 2317.407180] LNet: Removed LNI 192.168.202.17@tcp [ 2317.451980] LNet: Removed LNI 10.1.2.3@tcp [ 2318.060348] Key type .llcrypt unregistered [ 2318.065348] Key type ._llcrypt unregistered [ 2332.267346] Key type ._llcrypt registered [ 2332.268688] Key type .llcrypt registered [ 2333.771968] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:49:19 (1768556959) [ 2334.236377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2334.268610] alg: No test for adler32 (adler32-zlib) [ 2335.334115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.761659] LNet: 119794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.527510] Key type .llcrypt unregistered [ 2336.529409] Key type ._llcrypt unregistered [ 2342.618596] Key type ._llcrypt registered [ 2342.620921] Key type .llcrypt registered [ 2342.839879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2342.852465] alg: No test for adler32 (adler32-zlib) [ 2343.890990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2343.946055] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2352.135887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 2352.298673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2352.858810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.117@tcp [ 2353.063470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2353.088213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2353.119465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2353.149395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2353.173991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2353.186386] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.117@tcp [ 2353.198140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2353.218441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2353.235368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2353.246256] LNet: 120240:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.258813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2353.287969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2353.310803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2353.341271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2353.374958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2353.411788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2353.946924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2354.162803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2354.194573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2354.227822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2354.250542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2354.272820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2354.298818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2354.330506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2354.364450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2354.382231] LNet: 120357:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2354.401568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2354.422456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2354.451241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2354.475485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2354.502205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2354.528636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2355.170957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.117@tcp [ 2355.425516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2355.455977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2355.481588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2355.502417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2355.521359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2355.537852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2355.565216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2355.593731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2355.608553] LNet: 120474:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2355.644796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2355.684130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.709512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.734612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.759517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.787563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.255395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2356.467665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2356.484896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2356.514873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2356.546398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2356.576802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2356.613075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2356.654365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2356.684600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2356.712898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2356.769465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2356.823095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2356.859921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2356.888550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2356.915899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2357.224478] LNet: 120636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.230265] LNetError: 120636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.246699] LNet: Removed LNI 192.168.202.17@tcp [ 2357.623473] Key type .llcrypt unregistered [ 2357.629490] Key type ._llcrypt unregistered [ 2372.709723] Key type ._llcrypt registered [ 2372.712212] Key type .llcrypt registered [ 2373.842948] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:49:59 (1768556999) [ 2374.337052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.349865] alg: No test for adler32 (adler32-zlib) [ 2375.251883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.612706] LNet: 121520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.150914] Key type .llcrypt unregistered [ 2376.152595] Key type ._llcrypt unregistered [ 2381.616192] Key type ._llcrypt registered [ 2381.618545] Key type .llcrypt registered [ 2381.768791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2381.793589] alg: No test for adler32 (adler32-zlib) [ 2382.787732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2382.821471] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2390.211861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.117@tcp [ 2398.243621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.293775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2398.330306] LNet: Added LNI 192.168.202.17@tcp1 [8/256/0/180] [ 2399.371578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2400.091576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.117@tcp [ 2400.227308] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.117@tcp [ 2400.311091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2400.337243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2400.367381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2400.398727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2400.429833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2400.459876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2400.478129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2400.512198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2400.531816] LNet: 122167:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2400.557731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2400.585461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2400.618451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2400.655509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2400.708459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2400.747378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2401.339703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2401.578644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2401.614715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2401.644910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2401.677802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2401.708464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2401.736160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2401.768787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2401.794639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2401.810812] LNet: 122285:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2401.830809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2401.872848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2401.912246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2401.948721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2401.969326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2401.998889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2402.561496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.117@tcp [ 2402.780994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2402.805703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2402.846656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2402.868833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2402.892526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2402.920495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2402.954250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2402.986865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2403.011528] LNet: 122402:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2403.027140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2403.049835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2403.073793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2403.097756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2403.121848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2403.161409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2403.643593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2403.838692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2403.868996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2403.896332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2403.917370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2403.932865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2403.960991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2403.986725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2404.012776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2404.037178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2404.056955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2404.081050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2404.108336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2404.132208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2404.160155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2408.891636] LNet: 122599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.898666] LNetError: 122599:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2408.909316] LNet: Removed LNI 192.168.202.17@tcp [ 2408.925124] LNet: Removed LNI 192.168.202.17@tcp1 [ 2409.328098] Key type .llcrypt unregistered [ 2409.344492] Key type ._llcrypt unregistered [ 2424.218874] Key type ._llcrypt registered [ 2424.224877] Key type .llcrypt registered [ 2425.581862] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 04:50:51 (1768557051) [ 2431.762545] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 04:50:57 (1768557057) [ 2437.721950] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 04:51:04 (1768557064) [ 2444.349876] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 04:51:10 (1768557070) [ 2451.893307] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 04:51:17 (1768557077) [ 2459.979475] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 04:51:25 (1768557085) [ 2467.956917] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 04:51:34 (1768557094) [ 2474.614147] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 04:51:40 (1768557100) [ 2480.975570] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 04:51:46 (1768557106) [ 2486.569987] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 04:51:52 (1768557112) [ 2493.290699] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 04:51:59 (1768557119) [ 2498.837621] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 04:52:04 (1768557124) [ 2504.400560] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 04:52:10 (1768557130) [ 2512.487328] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 04:52:18 (1768557138) [ 2519.422708] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 04:52:25 (1768557145) [ 2526.991516] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 04:52:33 (1768557153) [ 2535.320539] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 04:52:41 (1768557161) [ 2543.265992] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 04:52:48 (1768557168) [ 2543.625529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.652453] alg: No test for adler32 (adler32-zlib) [ 2544.585166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2544.742442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2544.760308] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2546.399118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2546.436733] LNet: 133305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.452362] LNetError: 133305:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.476841] LNet: Removed LNI 192.168.202.17@tcp [ 2546.498533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2546.587535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2546.625237] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2554.290656] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 04:52:59 (1768557179) [ 2561.841851] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 04:53:07 (1768557187) [ 2570.488680] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 04:53:16 (1768557196) [ 2578.342121] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 04:53:24 (1768557204) [ 2584.721420] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 04:53:30 (1768557210) [ 2591.032220] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 04:53:37 (1768557217) [ 2598.097127] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 04:53:43 (1768557223) [ 2604.709336] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 04:53:51 (1768557231) [ 2612.396294] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 04:53:58 (1768557238) [ 2622.731772] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 04:54:08 (1768557248) [ 2630.028257] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 04:54:15 (1768557255) [ 2636.960827] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 04:54:23 (1768557263) [ 2645.473554] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2426 sec ========== 04:54:30 (1768557270) [ 2645.909679] LNet: 140771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.920972] LNetError: 140771:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2645.944816] LNet: Removed LNI 192.168.202.17@tcp [ 2646.406426] Key type .llcrypt unregistered [ 2646.408435] Key type ._llcrypt unregistered [ 2659.892786] Key type ._llcrypt registered [ 2659.894406] Key type .llcrypt registered [ 2660.572494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2660.599964] alg: No test for adler32 (adler32-zlib) [ 2661.683617] Lustre: Lustre: Build Version: 2.17.50_30_gc0571ea [ 2662.043033] LNet: Added LNI 192.168.202.17@tcp [8/256/0/180] [ 2663.887181] Key type lgssc registered [ 2665.077619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2739.766180] Lustre: Mounted lustre-client [ 2744.643978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2757.376444] LustreError: 143225:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff91034937a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2757.394306] LustreError: 143225:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2757.421947] Lustre: Unmounted lustre-client [ 2776.893875] Key type lgssc unregistered [ 2777.137444] LNet: 143706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2777.143519] LNetError: 143706:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2777.159866] LNet: Removed LNI 192.168.202.17@tcp [ 2777.856158] Key type .llcrypt unregistered [ 2777.857657] Key type ._llcrypt unregistered