[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 428478494 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002000] *** VALIDATE selinux *** [ 0.006612] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.010792] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.013120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.014101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.015110] *** VALIDATE tmpfs *** [ 0.017091] *** VALIDATE proc *** [ 0.018231] *** VALIDATE cgroup *** [ 0.019008] *** VALIDATE cgroup2 *** [ 0.020247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.021137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.022008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.023029] Spectre V2 : User space: Vulnerable [ 0.024009] Speculative Store Bypass: Vulnerable [ 0.028346] debug: unmapping init [mem 0xffffffffa1059000-0xffffffffa1060fff] [ 0.030322] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.031816] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.032024] ... version: 2 [ 0.033016] ... bit width: 48 [ 0.034012] ... generic registers: 4 [ 0.035016] ... value mask: 0000ffffffffffff [ 0.036012] ... max period: 00007fffffffffff [ 0.037012] ... fixed-purpose events: 3 [ 0.038010] ... event mask: 000000070000000f [ 0.039342] rcu: Hierarchical SRCU implementation. [ 0.041525] smp: Bringing up secondary CPUs ... [ 0.042596] x86: Booting SMP configuration: [ 0.043022] .... node #0, CPUs: #1 #2 #3 [ 0.054126] smp: Brought up 1 node, 4 CPUs [ 0.056012] smpboot: Max logical packages: 1 [ 0.057010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.086061] node 0 deferred pages initialised in 28ms [ 0.089776] devtmpfs: initialized [ 0.090224] x86/mm: Memory block size: 128MB [ 0.093057] gcov: version magic: 0x41383552 [ 0.095222] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.096070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.097230] pinctrl core: initialized pinctrl subsystem [ 0.098402] [ 0.098960] ************************************************************* [ 0.099012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.100013] ** ** [ 0.101012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.102011] ** ** [ 0.103014] ** This means that this kernel is built to expose internal ** [ 0.104013] ** IOMMU data structures, which may compromise security on ** [ 0.105012] ** your system. ** [ 0.106011] ** ** [ 0.107014] ** If you see this message and you are not debugging the ** [ 0.108015] ** kernel, report this immediately to your vendor! ** [ 0.109013] ** ** [ 0.110011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.111016] ************************************************************* [ 0.112726] NET: Registered protocol family 16 [ 0.113526] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.114059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.115066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.116707] cpuidle: using governor menu [ 0.119029] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.121648] PCI: Using configuration type 1 for base access [ 0.124126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.135048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.136015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.137537] cryptd: max_cpu_qlen set to 1000 [ 0.139000] ACPI: Added _OSI(Module Device) [ 0.139000] ACPI: Added _OSI(Processor Device) [ 0.142016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.143013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.148672] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.155841] ACPI: Interpreter enabled [ 0.156062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.157013] ACPI: Using IOAPIC for interrupt routing [ 0.158099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.159487] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.171756] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.174065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.176018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.180130] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.187613] acpiphp: Slot [2] registered [ 0.189141] acpiphp: Slot [5] registered [ 0.191106] acpiphp: Slot [6] registered [ 0.192097] acpiphp: Slot [3] registered [ 0.194148] acpiphp: Slot [4] registered [ 0.196180] acpiphp: Slot [7] registered [ 0.198153] acpiphp: Slot [8] registered [ 0.199085] acpiphp: Slot [9] registered [ 0.201144] acpiphp: Slot [10] registered [ 0.202148] acpiphp: Slot [11] registered [ 0.204093] acpiphp: Slot [12] registered [ 0.206148] acpiphp: Slot [13] registered [ 0.208140] acpiphp: Slot [14] registered [ 0.210091] acpiphp: Slot [15] registered [ 0.213256] acpiphp: Slot [16] registered [ 0.215086] acpiphp: Slot [17] registered [ 0.217090] acpiphp: Slot [18] registered [ 0.219125] acpiphp: Slot [19] registered [ 0.220178] acpiphp: Slot [20] registered [ 0.222094] acpiphp: Slot [21] registered [ 0.224098] acpiphp: Slot [22] registered [ 0.226125] acpiphp: Slot [23] registered [ 0.228084] acpiphp: Slot [24] registered [ 0.230092] acpiphp: Slot [25] registered [ 0.231081] acpiphp: Slot [26] registered [ 0.233089] acpiphp: Slot [27] registered [ 0.235102] acpiphp: Slot [28] registered [ 0.237102] acpiphp: Slot [29] registered [ 0.238092] acpiphp: Slot [30] registered [ 0.240096] acpiphp: Slot [31] registered [ 0.241065] PCI host bridge to bus 0000:00 [ 0.243020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.246021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.248019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.251022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.255041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.258022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.260178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.263054] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.267415] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.275012] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.280051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.284022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.287018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.290017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.292704] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.296720] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.299040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.302880] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.307010] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.315013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.319010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.323813] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.330015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.341018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.366019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.386036] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.395029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.400017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.418039] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.433910] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.437488] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.441383] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.445813] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.449227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.456040] iommu: Default domain type: Passthrough [ 0.458692] SCSI subsystem initialized [ 0.461134] ACPI: bus type USB registered [ 0.463244] usbcore: registered new interface driver usbfs [ 0.466085] usbcore: registered new interface driver hub [ 0.468075] usbcore: registered new device driver usb [ 0.470167] pps_core: LinuxPPS API ver. 1 registered [ 0.472012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.477054] PTP clock support registered [ 0.479091] EDAC MC: Ver: 3.0.0 [ 0.480365] PCI: Using ACPI for IRQ routing [ 0.481751] NetLabel: Initializing [ 0.482008] NetLabel: domain hash size = 128 [ 0.483008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.484080] NetLabel: unlabeled traffic allowed by default [ 0.486583] vgaarb: loaded [ 0.487332] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.489014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.495282] clocksource: Switched to clocksource kvm-clock [ 0.629525] VFS: Disk quotas dquot_6.6.0 [ 0.630807] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.633196] *** VALIDATE ramfs *** [ 0.634389] *** VALIDATE hugetlbfs *** [ 0.635767] pnp: PnP ACPI init [ 0.638296] pnp: PnP ACPI: found 6 devices [ 0.655442] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.658681] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.661080] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.663374] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.666032] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.668237] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.670909] NET: Registered protocol family 2 [ 0.673399] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.678243] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.681893] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.687153] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.690391] TCP: Hash tables configured (established 65536 bind 65536) [ 0.693253] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.696147] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.698941] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.702234] NET: Registered protocol family 1 [ 0.704622] RPC: Registered named UNIX socket transport module. [ 0.707099] RPC: Registered udp transport module. [ 0.709224] RPC: Registered tcp transport module. [ 0.711255] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.713899] NET: Registered protocol family 44 [ 0.715797] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.717755] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.720177] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.722350] PCI: CLS 0 bytes, default 64 [ 0.724179] Unpacking initramfs... [ 2.439832] debug: unmapping init [mem 0xffff9a967cc64000-0xffff9a967ffcffff] [ 2.443963] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.445914] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.448688] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.182444] Initialise system trusted keyrings [ 3.183840] Key type blacklist registered [ 3.185941] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.195649] zbud: loaded [ 3.198884] *** VALIDATE nfs *** [ 3.200018] *** VALIDATE nfs4 *** [ 3.201447] pstore: using deflate compression [ 3.204863] Platform Keyring initialized [ 3.309307] NET: Registered protocol family 38 [ 3.310783] Key type asymmetric registered [ 3.311889] Asymmetric key parser 'x509' registered [ 3.313634] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.316103] io scheduler mq-deadline registered [ 3.317351] io scheduler kyber registered [ 3.318764] io scheduler bfq registered [ 3.320684] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.323511] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.326191] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.328590] ACPI: Power Button [PWRF] [ 3.334145] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.340354] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.350131] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.378630] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.407807] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.412722] Non-volatile memory driver v1.3 [ 3.414565] Linux agpgart interface v0.103 [ 3.446918] virtio_blk virtio1: [vda] 139272 512-byte logical blocks (71.3 MB/68.0 MiB) [ 3.449941] vda: detected capacity change from 0 to 71307264 [ 3.510782] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.515830] vdb: detected capacity change from 0 to 1073741824 [ 3.611122] libphy: Fixed MDIO Bus: probed [ 3.647381] usbcore: registered new interface driver usbserial_generic [ 3.670449] usbserial: USB Serial support registered for generic [ 3.673099] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.683704] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.687426] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.692844] mousedev: PS/2 mouse device common for all mice [ 3.698413] rtc_cmos 00:05: RTC can wake from S4 [ 3.703311] rtc_cmos 00:05: registered as rtc0 [ 3.705020] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.708708] intel_pstate: CPU model not supported [ 3.708941] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.742381] hpet1: lost 1 rtc interrupts [ 3.749179] hid: raw HID events driver (C) Jiri Kosina [ 3.752318] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.755291] usbcore: registered new interface driver usbhid [ 3.755298] usbhid: USB HID core driver [ 3.755434] drop_monitor: Initializing network drop monitor service [ 3.755603] Initializing XFRM netlink socket [ 3.756727] NET: Registered protocol family 10 [ 3.761988] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.775835] Segment Routing with IPv6 [ 3.777334] NET: Registered protocol family 17 [ 3.779457] mpls_gso: MPLS GSO support [ 3.786227] RAS: Correctable Errors collector initialized. [ 3.788093] AVX version of gcm_enc/dec engaged. [ 3.789409] AES CTR mode by8 optimization enabled [ 3.893824] sched_clock: Marking stable (3893803475, 0)->(4965178290, -1071374815) [ 3.897617] registered taskstats version 1 [ 3.899522] Loading compiled-in X.509 certificates [ 3.901474] zswap: loaded using pool lzo/zbud [ 3.928729] Key type big_key registered [ 3.948726] Key type encrypted registered [ 3.950419] ima: No TPM chip found, activating TPM-bypass! [ 3.952687] ima: Allocated hash algorithm: sha1 [ 3.954611] ima: No architecture policies found [ 3.956484] evm: Initialising EVM extended attributes: [ 3.958511] evm: security.selinux [ 3.961421] evm: security.ima [ 3.962634] evm: security.capability [ 3.964145] evm: HMAC attrs: 0x1 [ 3.967400] rtc_cmos 00:05: setting system clock to 2026-06-18 21:14:55 UTC (1781817295) [ 3.976601] debug: unmapping init [mem 0xffffffffa2003000-0xffffffffa21fffff] [ 3.980952] debug: unmapping init [mem 0xffffffffa0d82000-0xffffffffa1058fff] [ 3.989672] Write protecting the kernel read-only data: 28672k [ 4.002428] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 4.010625] debug: unmapping init [mem 0xffffffff9fd14000-0xffffffff9fdfffff] [ 4.074450] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.080182] systemd[1]: Detected virtualization kvm. [ 4.081816] systemd[1]: Detected architecture x86-64. [ 4.083229] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.111938] systemd[1]: No hostname configured. [ 4.113072] systemd[1]: Set hostname to . [ 4.116323] random: systemd: uninitialized urandom read (16 bytes read) [ 4.119860] systemd[1]: Initializing machine ID from random generator. [ 4.184589] random: ln: uninitialized urandom read (6 bytes read) [ 4.379961] random: systemd: uninitialized urandom read (16 bytes read) [ 4.383296] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.393338] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.411216] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.341992] device-mapper: uevent: version 1.0.3 [ 6.345784] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 7.427138] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 7.606914] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.932461] scsi host0: ata_piix [ 7.971148] scsi host1: ata_piix [ 7.974159] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.988552] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 13.777208] random: crng init done [ 13.787491] random: 7 urandom warning(s) missed due to ratelimiting [ 17.316886] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.507625] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.542283] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.279629] SELinux: Disabled at runtime. [ 25.437444] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.452979] systemd[1]: Detected virtualization kvm. [ 25.456292] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.880276] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.884300] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.899292] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.905059] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.909676] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.917995] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.939312] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 27.751377] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 29.347462] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.601387] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.706914] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.445317] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.737645] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 35.675362] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.100973] NFS: Registering the id_resolver key type [ 37.110794] Key type id_resolver registered [ 37.119661] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg236-client login: [ 106.783390] libcfs: loading out-of-tree module taints kernel. [ 107.001381] Key type ._llcrypt registered [ 107.111320] Key type .llcrypt registered [ 107.482495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 107.493190] alg: No test for adler32 (adler32-zlib) [ 108.657416] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 109.268549] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 110.975161] Key type lgssc registered [ 112.117256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 202.360072] Lustre: Mounted lustre-client [ 205.900267] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 213.045899] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing check_logdir /tmp/testlogs/ [ 216.334963] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing yml_node [ 219.723908] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 221.791667] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 223.816503] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 224.970670] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jun 18 17:18:35 EDT 2026 [ 227.817302] Lustre: lustre-OST0000-osc-ffff9a96c5630800: disconnect after 24s idle [ 235.427810] Lustre: Unmounted lustre-client [ 269.097827] Key type lgssc unregistered [ 269.253550] LNet: 8375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.320590] LNet: Removed LNI 192.168.202.36@tcp [ 270.891125] Key type .llcrypt unregistered [ 270.895063] Key type ._llcrypt unregistered [ 290.364640] Key type ._llcrypt registered [ 290.367737] Key type .llcrypt registered [ 290.511433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 290.530426] alg: No test for adler32 (adler32-zlib) [ 291.467658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 291.532371] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 291.544496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 294.385626] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 294.463313] LNet: 9251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.477224] LNet: Removed LNI 192.168.202.36@tcp [ 294.960609] Key type .llcrypt unregistered [ 294.962380] Key type ._llcrypt unregistered [ 299.520207] Key type ._llcrypt registered [ 299.522745] Key type .llcrypt registered [ 300.633733] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:19:50 (1781817590) [ 301.036868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 301.060296] alg: No test for adler32 (adler32-zlib) [ 301.959301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.039563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.059411] LNet: 9668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.077697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.123293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 302.136312] LNet: 9675:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 307.523439] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:19:57 (1781817597) [ 307.595713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.616690] LNet: 10245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.637180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.706061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 312.668607] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:20:03 (1781817603) [ 312.832497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.853462] LNet: 10836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.871598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.942200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 317.461210] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:20:08 (1781817608) [ 317.548121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.581123] LNet: 11415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.606275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.686194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 322.433390] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:20:13 (1781817613) [ 322.651421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.670799] LNet: 12004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.685898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.743865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 322.991050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 327.762448] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:20:18 (1781817618) [ 328.033956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.089182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.170068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 328.599949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 329.072314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 333.919069] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:20:24 (1781817624) [ 334.202313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.220254] LNet: 13191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.223932] LNet: 13191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 334.240649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.297922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 339.084068] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:20:29 (1781817629) [ 339.209153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.259165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.302755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 339.326954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 339.553336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 339.571219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 339.830131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 339.856026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 340.145563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 340.168688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 340.438206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 340.458873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 340.700530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 340.725094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 340.966642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 341.002043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 341.279347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 341.303924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 341.675839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 341.699793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 341.936746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 341.958265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 342.266672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 342.294617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 342.495116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 342.512822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 342.800585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 342.820522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 347.555857] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:20:38 (1781817638) [ 347.743128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.770248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.825566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 347.862377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 352.597975] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:20:43 (1781817643) [ 352.694477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.722511] LNet: 15038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.727807] LNet: 15038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 352.750352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.826643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 352.841687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 357.511287] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:20:48 (1781817648) [ 357.581448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.622885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.689765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 357.720858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 361.995443] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:20:52 (1781817652) [ 362.144842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.173946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.230827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 362.264396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 366.248548] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:20:56 (1781817656) [ 366.363952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.406865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.469462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 366.491458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 371.375156] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:21:01 (1781817661) [ 371.579869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.604511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.675720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 371.703327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 376.012560] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:21:06 (1781817666) [ 376.230577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.253556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.341923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 376.692740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 381.730200] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:21:12 (1781817672) [ 381.821544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.884017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.095906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 382.253749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 387.086080] hrtimer: interrupt took 5465765 ns [ 387.172750] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:21:17 (1781817677) [ 387.217899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.234525] LNet: 19183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.240741] LNet: 19183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 387.257228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.340560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 387.501319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 392.357741] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:21:22 (1781817682) [ 392.424163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.458845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.775410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 392.914958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 397.231591] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:21:27 (1781817687) [ 397.372687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.416360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.471680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.509537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 401.688947] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:21:32 (1781817692) [ 401.950524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.981677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.044144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 402.074628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 406.317602] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:21:37 (1781817697) [ 406.614768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.650903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.738244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 411.401813] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:21:42 (1781817702) [ 411.626965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.652477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.719767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 416.392423] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:21:47 (1781817707) [ 416.582624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.614409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.669315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 416.884703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 421.458398] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:21:51 (1781817711) [ 421.656343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.686658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.726314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 421.753555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 426.132913] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:21:56 (1781817716) [ 426.421594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.454289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.517600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 426.544331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 431.735221] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:22:02 (1781817722) [ 431.961321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.003363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.063691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 432.096334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 437.164230] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:22:07 (1781817727) [ 437.411855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.436063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.509670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 437.541665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 442.047324] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:22:12 (1781817732) [ 442.187148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.231675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.263967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 442.324250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 442.342509] LNetError: 25771:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 442.375672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 442.398628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 446.701303] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:22:17 (1781817737) [ 446.746781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.784859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.863287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.48OWjcoDhu [ 446.906896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.48OWjcoDhu [ 446.953547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.48OWjcoDhu [ 451.172397] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:22:21 (1781817741) [ 451.417855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.431853] LNet: 26942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.434969] LNet: 26942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 451.448925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.484537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 451.507339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 451.547571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 455.883979] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:22:26 (1781817746) [ 456.005584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.049849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.092495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 458.028143] LNet: Message forwarding will be enabled in 2s [ 461.087143] LNet: Message forwarding enabled [ 461.394616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 461.412638] LNet: Message forwarding will stop in 2s to 6s [ 465.777773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 466.210851] LNet: Message forwarding disabled [ 467.161953] LNet: Message forwarding will be enabled in 2s [ 470.303596] LNet: Message forwarding enabled [ 471.779923] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:22:42 (1781817762) [ 471.986973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.647271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.712541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 476.756942] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:22:47 (1781817767) [ 476.812565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.840557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.873511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 476.903463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 476.927488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 476.956522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 476.996465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 477.025758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 477.058806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 477.086882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 477.114670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 477.140941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 477.167945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 477.196954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 477.223741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 477.250489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 477.279696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 477.311769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 477.351100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 477.389505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 477.422428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 477.441798] LNetError: 28840:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 482.005463] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:22:52 (1781817772) [ 482.228984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.270936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.336612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.556434] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:22:57 (1781817777) [ 487.877857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg236-server [ 488.949759] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:22:59 (1781817779) [ 489.146741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.162976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.193382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 489.218620] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 490.296634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[37-41]@tcp [ 490.314524] LNet: 30555:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 490.358135] LNetError: 30345:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.37@tcp is being used as a gateway but routing feature is not turned on [ 490.370213] LNetError: 30345:0:(lib-lnet.h:1234:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.37@tcp has gone from up to down [ 491.383386] LNetError: 30345:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.40@tcp is being used as a gateway but routing feature is not turned on [ 491.397943] LNetError: 30345:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 492.390720] LNetError: 30345:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.37@tcp is being used as a gateway but routing feature is not turned on [ 492.396700] LNetError: 30345:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 493.407277] LNetError: 30352:0:(lib-lnet.h:1234:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.39@tcp has gone from up to down [ 493.418438] LNetError: 30352:0:(lib-lnet.h:1234:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 494.437059] LNetError: 30345:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.40@tcp is being used as a gateway but routing feature is not turned on [ 494.442137] LNetError: 30345:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 494.762181] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:23:05 (1781817785) [ 495.974299] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg236-server [ 497.059766] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:23:07 (1781817787) [ 497.180745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.281208] LNet: Removed LNI 192.168.202.36@tcp [ 498.305428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.344650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 498.361771] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 498.885818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[37-42/2]@tcp [ 498.904607] LNet: 31607:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 498.910606] LNet: 31607:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 498.926831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[37-42/2]@tcp [ 498.933621] LNetError: 31494:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.37@tcp is being used as a gateway but routing feature is not turned on [ 498.949457] LNetError: 31494:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 8 previous similar messages [ 498.956702] LNetError: 31494:0:(lib-lnet.h:1234:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.37@tcp has gone from up to down [ 498.967632] LNetError: 31494:0:(lib-lnet.h:1234:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 503.723355] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:23:14 (1781817794) [ 503.886532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.941274] LNet: Removed LNI 192.168.202.36@tcp [ 505.001374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.131343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 505.168490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.192784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.244825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.286125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.338467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.387402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 505.467954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.495623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.558375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.629353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.657489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.702819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 505.774482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.816416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.860995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.921624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.957698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.010708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 506.063677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.095359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.151689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.202736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.231607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.269369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 506.337111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.368250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.411127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.474511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.508581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.545608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 506.617661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.658875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.709763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.061351] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:23:21 (1781817801) [ 512.333120] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg236-server [ 513.495303] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:23:24 (1781817804) [ 514.577446] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg236-server [ 516.042626] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:23:26 (1781817806) [ 516.724606] Key type .llcrypt unregistered [ 516.725796] Key type ._llcrypt unregistered [ 521.057308] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 521.136736] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 521.159789] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 521.169529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 523.641389] Key type ._llcrypt registered [ 523.644408] Key type .llcrypt registered [ 523.777706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 523.788378] alg: No test for adler32 (adler32-zlib) [ 524.705737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.745869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 524.759321] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 525.194317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 525.214846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 526.601796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 526.621483] LNet: Removed LNI 10.1.2.3@tcp [ 526.714312] LNet: 34320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.730891] LNet: Removed LNI 192.168.202.36@tcp [ 527.181284] Key type .llcrypt unregistered [ 527.183097] Key type ._llcrypt unregistered [ 534.678632] Key type ._llcrypt registered [ 534.680290] Key type .llcrypt registered [ 535.590527] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:23:46 (1781817826) [ 536.005405] Key type .llcrypt unregistered [ 536.007202] Key type ._llcrypt unregistered [ 540.356181] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 540.472060] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 540.479389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 540.495118] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 543.022676] Key type ._llcrypt registered [ 543.025253] Key type .llcrypt registered [ 543.188104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 543.198220] alg: No test for adler32 (adler32-zlib) [ 544.085761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.139054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 544.170293] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 544.649821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 544.669344] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 546.498344] LNet: 35813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.502348] LNet: Removed LNI 192.168.202.36@tcp [ 546.509356] LNet: Removed LNI 10.1.2.3@tcp [ 546.986420] Key type .llcrypt unregistered [ 546.990532] Key type ._llcrypt unregistered [ 555.474482] Key type ._llcrypt registered [ 555.476519] Key type .llcrypt registered [ 556.318293] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:24:06 (1781817846) [ 556.604681] Key type .llcrypt unregistered [ 556.609930] Key type ._llcrypt unregistered [ 560.644368] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 560.763516] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 560.766043] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 560.768785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 563.092890] Key type ._llcrypt registered [ 563.095919] Key type .llcrypt registered [ 563.218872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.229615] alg: No test for adler32 (adler32-zlib) [ 564.129131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.175722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 564.190800] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 565.337475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 565.357026] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 566.650714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 566.677881] LNet: Removed LNI 10.1.2.3@tcp [ 566.691342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 566.720741] LNet: Removed LNI 10.1.2.31@tcp [ 566.732364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 566.751517] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 567.165088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 567.187137] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 567.809342] LNet: 37728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.812718] LNet: Removed LNI 10.1.2.3@tcp [ 567.822742] LNet: Removed LNI 10.1.2.31@tcp [ 568.166654] Key type .llcrypt unregistered [ 568.168906] Key type ._llcrypt unregistered [ 574.987730] Key type ._llcrypt registered [ 574.988706] Key type .llcrypt registered [ 575.847694] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:24:26 (1781817866) [ 576.086605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 576.097859] alg: No test for adler32 (adler32-zlib) [ 576.961514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.009814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 577.023734] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 577.550565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.569234] LNet: 38680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.584920] LNet: Removed LNI 192.168.202.36@tcp [ 577.597476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.642726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.36@tcp [ 577.657220] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 578.158701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.36@tcp [ 578.182351] LNet: Removed LNI 192.168.202.36@tcp [ 578.234860] LNet: 38815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.691576] Key type .llcrypt unregistered [ 578.693311] Key type ._llcrypt unregistered [ 586.303766] Key type ._llcrypt registered [ 586.306247] Key type .llcrypt registered [ 587.140196] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:24:37 (1781817877) [ 588.337516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg236-server [ 589.453108] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:24:40 (1781817880) [ 589.713637] Key type .llcrypt unregistered [ 589.716337] Key type ._llcrypt unregistered [ 593.073496] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 593.110270] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 593.113215] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 593.115600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 595.382073] Key type ._llcrypt registered [ 595.383204] Key type .llcrypt registered [ 595.449755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 595.455982] alg: No test for adler32 (adler32-zlib) [ 596.291795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.321829] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 596.675571] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 597.645300] LNet: 40650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.647637] LNet: Removed LNI 192.168.202.36@tcp1 [ 597.657846] LNet: Removed LNI 10.1.2.3@tcp2 [ 597.942840] Key type .llcrypt unregistered [ 597.944334] Key type ._llcrypt unregistered [ 603.741796] Key type ._llcrypt registered [ 603.742811] Key type .llcrypt registered [ 604.431439] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:24:55 (1781817895) [ 604.700724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 604.715107] alg: No test for adler32 (adler32-zlib) [ 605.561171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.596844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 605.611985] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 606.170427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.181686] LNet: 41617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.195568] LNet: Removed LNI 192.168.202.36@tcp [ 606.209660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.259656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 606.275837] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 614.777913] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:25:05 (1781817905) [ 614.885455] LNet: 42207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.905674] LNet: Removed LNI 192.168.202.36@tcp [ 615.298692] Key type .llcrypt unregistered [ 615.300845] Key type ._llcrypt unregistered [ 618.526279] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 618.632512] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 618.640052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 618.643346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 620.988850] Key type ._llcrypt registered [ 620.990984] Key type .llcrypt registered [ 621.107645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 621.117249] alg: No test for adler32 (adler32-zlib) [ 621.984698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.042952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.064133] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 622.539880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 622.557022] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 623.884198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.903560] LNet: 42944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.908217] LNet: Removed LNI 192.168.202.36@tcp [ 623.920587] LNet: Removed LNI 10.1.2.3@tcp [ 623.931663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.978617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 623.995275] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 623.997668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 624.313586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.336614] LNet: Removed LNI 192.168.202.36@tcp [ 624.351572] LNet: Removed LNI 10.1.2.3@tcp [ 624.362838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.405030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 624.419806] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 624.422429] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 624.691623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 624.711125] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 625.315934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.329574] LNet: 43134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.333130] LNet: 43134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 625.337430] LNet: Removed LNI 192.168.202.36@tcp [ 625.339866] LNet: Removed LNI 10.1.2.3@tcp [ 625.347151] LNet: Removed LNI 192.168.202.36@tcp2 [ 625.375383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.423592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 625.439372] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 625.441333] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 625.444424] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 629.538983] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:25:20 (1781817920) [ 629.700667] LNet: 43731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.703543] LNet: Removed LNI 192.168.202.36@tcp [ 629.706546] LNet: Removed LNI 10.1.2.3@tcp [ 629.719889] LNet: Removed LNI 192.168.202.36@tcp2 [ 630.156750] Key type .llcrypt unregistered [ 630.158428] Key type ._llcrypt unregistered [ 634.012553] Key type ._llcrypt registered [ 634.013885] Key type .llcrypt registered [ 634.139134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 634.151490] alg: No test for adler32 (adler32-zlib) [ 635.036835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.094987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.129221] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 635.672397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.685831] LNet: 44124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.702989] LNet: Removed LNI 192.168.202.36@tcp [ 635.721641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.770292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 635.792315] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 636.068053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.088054] LNet: Removed LNI 192.168.202.36@tcp [ 636.101071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.136758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 636.154268] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 636.473476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.501859] LNet: 44185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.506434] LNet: 44185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 636.517980] LNet: Removed LNI 192.168.202.36@tcp [ 636.538193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.597079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 636.624387] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 636.901231] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 636.950438] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 636.955426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 636.958114] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 639.248208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.262516] LNet: 44263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.273506] LNet: Removed LNI 192.168.202.36@tcp [ 639.284431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.324707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 639.339789] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 643.051491] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:25:33 (1781817933) [ 643.183386] LNet: 44861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.193979] LNet: Removed LNI 192.168.202.36@tcp [ 643.588989] Key type .llcrypt unregistered [ 643.591449] Key type ._llcrypt unregistered [ 647.339388] Key type ._llcrypt registered [ 647.341385] Key type .llcrypt registered [ 647.476398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 647.486633] alg: No test for adler32 (adler32-zlib) [ 648.366206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.410984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.430689] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 649.022652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.035543] LNet: 45258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.046525] LNet: Removed LNI 192.168.202.36@tcp [ 649.068812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.118084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 649.133191] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 653.321509] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 17:25:44 (1781817944) [ 653.521641] LNet: 45850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.532690] LNet: Removed LNI 192.168.202.36@tcp [ 653.915540] Key type .llcrypt unregistered [ 653.919610] Key type ._llcrypt unregistered [ 657.917550] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 657.996063] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 658.002207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 658.010348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 660.380991] Key type ._llcrypt registered [ 660.386875] Key type .llcrypt registered [ 660.507446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 660.519558] alg: No test for adler32 (adler32-zlib) [ 661.351061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.386708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.400628] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 661.856694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.871808] LNet: 46302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.881864] LNet: Removed LNI 192.168.202.36@tcp [ 661.897098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.949955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 661.984974] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 666.098144] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 17:25:56 (1781817956) [ 666.185966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.207781] LNet: 46901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.220910] LNet: Removed LNI 192.168.202.36@tcp [ 666.237388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.303522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 669.840808] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 17:26:00 (1781817960) [ 670.024104] LNet: 47483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.304496] Key type .llcrypt unregistered [ 670.305616] Key type ._llcrypt unregistered [ 672.881307] Key type ._llcrypt registered [ 672.882950] Key type .llcrypt registered [ 672.946157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 672.953416] alg: No test for adler32 (adler32-zlib) [ 673.764729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.785414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 673.796720] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 674.108552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.118055] LNet: 47880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.125306] LNet: Removed LNI 192.168.202.36@tcp [ 674.133295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.160479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 674.174390] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 676.489529] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 17:26:07 (1781817967) [ 676.553892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.564266] LNet: 48473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.575433] LNet: Removed LNI 192.168.202.36@tcp [ 676.583803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.614684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 678.948071] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 17:26:09 (1781817969) [ 679.511202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 680.113539] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 17:26:11 (1781817971) [ 680.690806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 681.288915] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 17:26:12 (1781817972) [ 681.860723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 682.484148] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 17:26:13 (1781817973) [ 683.088761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 683.702337] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 17:26:14 (1781817974) [ 684.236582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 684.799632] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 17:26:15 (1781817975) [ 684.865654] LNet: 50521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.093454] Key type .llcrypt unregistered [ 685.094731] Key type ._llcrypt unregistered [ 687.032893] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.090320] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.092551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.094967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 689.265490] Key type ._llcrypt registered [ 689.266947] Key type .llcrypt registered [ 689.329478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 689.335902] alg: No test for adler32 (adler32-zlib) [ 690.140504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.168593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 690.178516] LNet: Added LNI 192.168.202.36@tcp [8/512/0/180] [ 690.181285] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 690.182795] LNet: 50838:0:(api-ni.c:2843:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 690.225178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.234710] LNet: 50857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.237619] LNet: Removed LNI 192.168.202.36@tcp [ 690.244384] LNet: Removed LNI 10.1.2.3@tcp [ 690.253647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.278020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 690.287414] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 690.309737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.319465] LNet: 50881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.332373] LNet: Removed LNI 192.168.202.36@tcp [ 690.340944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.365660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 690.378885] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 690.380904] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 690.382434] LNet: 50889:0:(api-ni.c:2843:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 690.419861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.429609] LNet: 50907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.433203] LNet: Removed LNI 192.168.202.36@tcp [ 690.447507] LNet: Removed LNI 10.1.2.3@tcp [ 690.456566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.487160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 690.495710] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 690.497687] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 690.499650] LNet: 50915:0:(api-ni.c:2843:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 690.570191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.580041] LNet: 50941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.583539] LNet: Removed LNI 192.168.202.36@tcp [ 690.596461] LNet: Removed LNI 10.1.2.3@tcp [ 690.605365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.634408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 690.643787] LNet: Added LNI 192.168.202.36@tcp [8/256/0/182] [ 690.646528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 690.684066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.691327] LNet: 50969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.693333] LNet: Removed LNI 192.168.202.36@tcp [ 690.700313] LNet: Removed LNI 10.1.2.3@tcp [ 690.707860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.731874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 690.744314] LNet: Added LNI 192.168.202.36@tcp [8/256/0/182] [ 690.746303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 690.747828] LNet: 50977:0:(api-ni.c:2843:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 690.776256] LNet: Removed LNI 192.168.202.36@tcp [ 690.781506] LNet: Removed LNI 10.1.2.3@tcp [ 690.988712] Key type .llcrypt unregistered [ 690.989663] Key type ._llcrypt unregistered [ 694.681986] Key type ._llcrypt registered [ 694.683577] Key type .llcrypt registered [ 695.212404] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:26:26 (1781817986) [ 695.346387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 695.353233] alg: No test for adler32 (adler32-zlib) [ 696.168735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.195412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.208111] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 696.563425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.36 [ 696.574725] LNet: 51952:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 696.579780] LNet: Removed LNI 192.168.202.36@tcp [ 699.034076] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:26:29 (1781817989) [ 699.063445] LNet: 52514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.074481] LNet: Removed LNI 192.168.202.36@tcp [ 699.336466] Key type .llcrypt unregistered [ 699.340676] Key type ._llcrypt unregistered [ 701.821340] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 701.949529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 704.326520] Key type ._llcrypt registered [ 704.327617] Key type .llcrypt registered [ 704.392026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 704.398467] alg: No test for adler32 (adler32-zlib) [ 705.388024] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 707.982032] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 17:26:38 (1781817998) [ 708.069739] LNet: 53433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.084927] LNet: Removed LNI 10.1.2.3@tcp [ 708.381905] Key type .llcrypt unregistered [ 708.384180] Key type ._llcrypt unregistered [ 710.757750] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 710.860781] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 710.863463] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 710.866269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 713.193304] Key type ._llcrypt registered [ 713.194606] Key type .llcrypt registered [ 713.252028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.260665] alg: No test for adler32 (adler32-zlib) [ 714.093097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 716.595970] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 17:26:47 (1781818007) [ 717.170130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 717.798873] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 17:26:48 (1781818008) [ 717.858253] LNet: 54611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.867420] LNet: Removed LNI 10.1.2.3@tcp [ 718.113540] Key type .llcrypt unregistered [ 718.115232] Key type ._llcrypt unregistered [ 720.416541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 720.530408] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 720.531722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 720.533185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 722.813872] Key type ._llcrypt registered [ 722.815236] Key type .llcrypt registered [ 722.888875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 722.895247] alg: No test for adler32 (adler32-zlib) [ 723.704743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.733394] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 726.713342] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 17:26:57 (1781818017) [ 726.879913] LNet: 55842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.894587] LNet: Removed LNI 10.1.2.3@tcp [ 727.120481] Key type .llcrypt unregistered [ 727.122203] Key type ._llcrypt unregistered [ 729.291213] Key type ._llcrypt registered [ 729.292695] Key type .llcrypt registered [ 729.363861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 729.370406] alg: No test for adler32 (adler32-zlib) [ 730.189434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 730.206178] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 733.550832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 733.805335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_interrupt -r 1 [ 733.824190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_interrupt -r 1 [ 733.843685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 733.853042] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 733.942573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 733.957792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 733.972240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 733.988163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 734.003457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 734.017718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 734.033540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 734.047894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 734.057915] LNet: 56279:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 734.066486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 734.082815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 734.097707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 734.111537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 734.128427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 734.144414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 734.159398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 734.365911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_dropped -r 1 [ 734.385300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_dropped -r 1 [ 734.402827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 734.413397] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 734.498923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 734.513616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 734.529397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 734.545280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 734.559974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 734.574623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 734.589627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 734.603672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 734.613179] LNet: 56388:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 734.622648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 734.637817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 734.657866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 734.671305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 734.685546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 734.699912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 734.714633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 734.924847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_aborted -r 1 [ 734.941398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_aborted -r 1 [ 734.956775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 735.049690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 735.065709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 735.079706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 735.093293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 735.108775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 735.123217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 735.137546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 735.152037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 735.165286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 735.180222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 735.194485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 735.208557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 735.231391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 735.248320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 735.264092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 735.467777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_no_route -r 1 [ 735.484508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_no_route -r 1 [ 735.500680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 735.511984] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 735.515098] LNet: Skipped 1 previous similar message [ 735.605354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 735.621556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 735.637151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 735.651738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 735.665539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 735.676926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 735.689746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 735.704234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 735.713711] LNet: 56606:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 735.716286] LNet: 56606:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 735.723835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 735.741344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 735.756684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 735.770779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 735.785679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 735.802539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 735.820085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 736.021371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_timeout -r 1 [ 736.039566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_timeout -r 1 [ 736.055110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 736.140172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 736.152232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 736.164260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 736.177476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 736.191822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 736.204782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 736.219636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 736.233813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 736.249287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 736.263100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 736.276637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 736.291413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 736.308428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 736.321750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 736.337856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 736.525361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_error -r 1 [ 736.541915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_error -r 1 [ 736.559048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 736.652863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 736.666595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 736.678923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 736.693590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 736.706978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 736.720901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 736.736701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 736.753940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 736.768869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 736.784321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 736.800833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 736.813599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 736.831214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 736.846581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 736.864658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 736.970846] LNet: 56865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.021274] LNet: Removed LNI 192.168.202.36@tcp [ 738.171420] Key type .llcrypt unregistered [ 738.172858] Key type ._llcrypt unregistered [ 743.764233] Key type ._llcrypt registered [ 743.765355] Key type .llcrypt registered [ 744.232958] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 17:27:15 (1781818035) [ 744.525348] Key type .llcrypt unregistered [ 744.526869] Key type ._llcrypt unregistered [ 746.765675] Key type ._llcrypt registered [ 746.766968] Key type .llcrypt registered [ 746.838952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 746.849056] alg: No test for adler32 (adler32-zlib) [ 747.660572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 747.673819] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 750.890527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 754.346618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.362058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 754.371873] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 754.842718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_interrupt -r 1 [ 754.854969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_interrupt -r 1 [ 754.868199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_interrupt -r 1 [ 754.883128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_interrupt -r 1 [ 754.899245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_interrupt -r 1 [ 754.915159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_interrupt -r 1 [ 754.929843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_interrupt -r 1 [ 754.944752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_interrupt -r 1 [ 754.958721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 754.967139] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 754.968973] LNet: 1 local NIs in recovery (showing 1): 192.168.202.36@tcp [ 755.042880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 755.056655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 755.070430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 755.083277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 755.095358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 755.107682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 755.120362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 755.134671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 755.142862] LNet: 58327:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 755.150080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 755.163210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 755.177427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 755.189872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 755.205252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 755.221567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 755.237874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 755.414995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_dropped -r 1 [ 755.431985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_dropped -r 1 [ 755.447384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_dropped -r 1 [ 755.461605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_dropped -r 1 [ 755.478410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_dropped -r 1 [ 755.491795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_dropped -r 1 [ 755.507173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_dropped -r 1 [ 755.522202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_dropped -r 1 [ 755.538906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 755.548533] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 755.551426] LNet: Skipped 2 previous similar messages [ 755.626519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 755.639596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 755.653075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 755.668379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 755.683823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 755.697932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 755.712931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 755.730741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 755.739183] LNet: 58448:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 755.747169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 755.758336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 755.770385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 755.783674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 755.802059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 755.816875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 755.831648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 756.031000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_aborted -r 1 [ 756.048351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_aborted -r 1 [ 756.065288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_aborted -r 1 [ 756.081288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_aborted -r 1 [ 756.095714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_aborted -r 1 [ 756.110646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_aborted -r 1 [ 756.125518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_aborted -r 1 [ 756.141038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_aborted -r 1 [ 756.156667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 756.240436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 756.254248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 756.267079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 756.281616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 756.295050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 756.307870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 756.320659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 756.334235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 756.346745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 756.359711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 756.372808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 756.385354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 756.399949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 756.413961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 756.428404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 756.610156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_no_route -r 1 [ 756.627479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_no_route -r 1 [ 756.643052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_no_route -r 1 [ 756.658428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_no_route -r 1 [ 756.675842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_no_route -r 1 [ 756.692246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_no_route -r 1 [ 756.707418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_no_route -r 1 [ 756.722876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_no_route -r 1 [ 756.739567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 756.749985] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 756.752719] LNet: Skipped 7 previous similar messages [ 756.835408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 756.852625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 756.870386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 756.888498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 756.901523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 756.916576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 756.929468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 756.942667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 756.950261] LNet: 58690:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 756.953212] LNet: 58690:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 756.960640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 756.975083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 756.987983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 757.001425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 757.016697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 757.032758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 757.045665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 757.238969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_timeout -r 1 [ 757.255495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_timeout -r 1 [ 757.271647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_timeout -r 1 [ 757.288676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_timeout -r 1 [ 757.308927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_timeout -r 1 [ 757.327485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_timeout -r 1 [ 757.344388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_timeout -r 1 [ 757.361167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_timeout -r 1 [ 757.380337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 757.468540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 757.481664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 757.500914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 757.516271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 757.530533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 757.546158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 757.560183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 757.573530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 757.587463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 757.602964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 757.617775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 757.630988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 757.646214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 757.661434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 757.678681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 757.868867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_error -r 1 [ 757.886454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_error -r 1 [ 757.902781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_error -r 1 [ 757.919912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_error -r 1 [ 757.936485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_error -r 1 [ 757.953666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_error -r 1 [ 757.969246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_error -r 1 [ 757.984739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_error -r 1 [ 757.999820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 758.088247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 758.100681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 758.112674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 758.125339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 758.139493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 758.153173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 758.172651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 758.189411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 758.204405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 758.219880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 758.235626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 758.250946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 758.268160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 758.284831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 758.300649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 760.554754] LNet: 59007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.567176] LNet: Removed LNI 192.168.202.36@tcp [ 762.597416] LNet: Removed LNI 192.168.202.36@tcp1 [ 762.742521] Key type .llcrypt unregistered [ 762.745043] Key type ._llcrypt unregistered [ 768.558237] Key type ._llcrypt registered [ 768.559135] Key type .llcrypt registered [ 769.062545] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 17:27:40 (1781818060) [ 769.398284] Key type .llcrypt unregistered [ 769.399433] Key type ._llcrypt unregistered [ 771.506854] Key type ._llcrypt registered [ 771.508170] Key type .llcrypt registered [ 771.579911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.585753] alg: No test for adler32 (adler32-zlib) [ 772.398793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 772.409727] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 775.530957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 775.760511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e remote_dropped -r 1 [ 775.776527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e remote_dropped -r 1 [ 775.792905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 775.800580] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 775.880042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 775.895319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 775.909983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 775.923118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 775.934808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 775.946685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 775.962170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 775.975175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 775.983083] LNet: 60253:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 775.991469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 776.004216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 776.019231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 776.033540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 776.047646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 776.069825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 776.092570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 776.289406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e remote_error -r 1 [ 776.309050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e remote_error -r 1 [ 776.328592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 776.339350] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 776.443500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 776.464175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 776.477878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 776.492593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 776.509124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 776.523550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 776.537380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 776.550162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 776.557706] LNet: 60362:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 776.565776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 776.580913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 776.595827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 776.612381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 776.630462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 776.646357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 776.659957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 776.855714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e remote_timeout -r 1 [ 776.872139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e remote_timeout -r 1 [ 776.886824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 776.969037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 776.983153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 777.000104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 777.015114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 777.029517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 777.042903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 777.055915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 777.069987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 777.081910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 777.095895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 777.113293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 777.128441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 777.145057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 777.159661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 777.174977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 777.281074] LNet: 60513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.345295] LNet: Removed LNI 192.168.202.36@tcp [ 778.504453] Key type .llcrypt unregistered [ 778.505606] Key type ._llcrypt unregistered [ 784.255232] Key type ._llcrypt registered [ 784.256842] Key type .llcrypt registered [ 784.776305] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 17:27:55 (1781818075) [ 785.089328] Key type .llcrypt unregistered [ 785.090821] Key type ._llcrypt unregistered [ 787.309124] Key type ._llcrypt registered [ 787.310183] Key type .llcrypt registered [ 787.389463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 787.398842] alg: No test for adler32 (adler32-zlib) [ 788.213108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 788.226430] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 791.523510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 794.940422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.956486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 794.968547] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 795.531147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e remote_dropped -r 1 [ 795.547904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e remote_dropped -r 1 [ 795.566540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e remote_dropped -r 1 [ 795.584763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e remote_dropped -r 1 [ 795.602562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e remote_dropped -r 1 [ 795.619275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e remote_dropped -r 1 [ 795.636551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e remote_dropped -r 1 [ 795.650703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e remote_dropped -r 1 [ 795.666151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 795.675772] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 795.678160] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.136@tcp [ 795.756351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 795.769125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 795.783192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 795.797217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 795.813103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 795.829923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 795.844721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 795.862696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 795.870773] LNet: 61975:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 795.879605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 795.896552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 795.913437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 795.928915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 795.943303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 795.960665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 795.974521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 795.988681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 796.177761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e remote_error -r 1 [ 796.193596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e remote_error -r 1 [ 796.209290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e remote_error -r 1 [ 796.226769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e remote_error -r 1 [ 796.247201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e remote_error -r 1 [ 796.265375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e remote_error -r 1 [ 796.281213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e remote_error -r 1 [ 796.295581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e remote_error -r 1 [ 796.310540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 796.320203] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 796.322996] LNet: Skipped 2 previous similar messages [ 796.399153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 796.411683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 796.427917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 796.442108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 796.460117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 796.476773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 796.490803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 796.504452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 796.512740] LNet: 62098:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 796.522316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 796.536536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 796.549240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 796.564536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 796.581724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 796.594890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 796.608368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 796.622462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 796.811802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e remote_timeout -r 1 [ 796.826708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e remote_timeout -r 1 [ 796.841957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e remote_timeout -r 1 [ 796.859768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e remote_timeout -r 1 [ 796.876396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e remote_timeout -r 1 [ 796.892393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e remote_timeout -r 1 [ 796.907679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e remote_timeout -r 1 [ 796.922705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e remote_timeout -r 1 [ 796.942895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.136@tcp [ 797.036398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 797.050292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 797.065993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 797.079739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 797.093966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 797.106425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 797.119202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 797.133714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 797.149511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 797.163369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 797.177198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 797.191474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 797.206603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 797.221519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 797.235698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 797.249149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 799.712456] LNet: 62298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.735239] LNet: Removed LNI 192.168.202.36@tcp [ 800.743871] LNet: Removed LNI 192.168.202.36@tcp1 [ 800.919569] Key type .llcrypt unregistered [ 800.921396] Key type ._llcrypt unregistered [ 806.816724] Key type ._llcrypt registered [ 806.818091] Key type .llcrypt registered [ 807.309068] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:28:18 (1781818098) [ 807.838557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 808.448856] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:28:19 (1781818099) [ 808.753270] Key type .llcrypt unregistered [ 808.754800] Key type ._llcrypt unregistered [ 810.973196] Key type ._llcrypt registered [ 810.974228] Key type .llcrypt registered [ 811.043302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 811.049198] alg: No test for adler32 (adler32-zlib) [ 811.855435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 811.868200] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 814.961527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 815.176268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e network_timeout -r 1 [ 815.192556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e network_timeout -r 1 [ 815.211149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 815.219040] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 815.297585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 815.309778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 815.321930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 815.334694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 815.348451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 815.365332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 815.383735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 815.398595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 815.406485] LNet: 63838:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 815.417331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 815.432640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 815.446990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 815.461861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 815.475382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 815.493491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 815.596400] LNet: 63877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.613395] LNet: Removed LNI 192.168.202.36@tcp [ 816.763471] Key type .llcrypt unregistered [ 816.765226] Key type ._llcrypt unregistered [ 823.151868] Key type ._llcrypt registered [ 823.153155] Key type .llcrypt registered [ 823.240521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.249862] alg: No test for adler32 (adler32-zlib) [ 824.051060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 824.061187] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 827.251970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 830.292833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.306297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 830.315736] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 830.809929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e network_timeout -r 1 [ 830.824763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e network_timeout -r 1 [ 830.839789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e network_timeout -r 1 [ 830.855446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e network_timeout -r 1 [ 830.871162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e network_timeout -r 1 [ 830.887292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e network_timeout -r 1 [ 830.902120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e network_timeout -r 1 [ 830.917338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e network_timeout -r 1 [ 830.934532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 830.945163] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 831.019420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 831.032137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 831.046848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 831.062270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 831.076764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 831.090605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 831.103544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 831.117619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 831.126890] LNet: 64752:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 831.135424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 831.150071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 831.166585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 831.182835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 831.197213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 831.211944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 833.478402] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 833.572419] LNet: 64825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.591145] LNet: Removed LNI 192.168.202.36@tcp [ 834.598417] LNet: Removed LNI 192.168.202.36@tcp1 [ 834.746985] Key type .llcrypt unregistered [ 834.748470] Key type ._llcrypt unregistered [ 840.560632] Key type ._llcrypt registered [ 840.561747] Key type .llcrypt registered [ 841.100069] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:28:52 (1781818132) [ 841.237139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.243887] alg: No test for adler32 (adler32-zlib) [ 842.060784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.086839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.098158] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 842.368697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 842.377785] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 842.703127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 842.729245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 842.748121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 842.763783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 842.779074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 842.793269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 842.808165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.36@tcp [ 842.818941] LNetError: 65923:0:(api-ni.c:4295:lnet_ni_set_healthv()) manually adding local NI 192.168.202.36@tcp to recovery [ 843.103260] LNet: 1 local NIs in recovery (showing 1): 192.168.202.36@tcp [ 843.103282] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 845.151392] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 846.001910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 849.247323] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 850.203251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 850.217250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 850.239617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.249335] LNet: 66040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.295130] LNet: Removed LNI 192.168.202.36@tcp [ 851.303314] LNet: Removed LNI 192.168.202.36@tcp1 [ 851.311667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.333643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.344095] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 851.637988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 851.647900] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 851.970474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 851.987896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 852.012911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 852.032675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 852.048022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 852.062366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 852.075883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 852.089428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.36@tcp [ 852.098996] LNetError: 66312:0:(api-ni.c:4295:lnet_ni_set_healthv()) manually adding local NI 192.168.202.36@tcp to recovery [ 852.383353] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 855.256019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 858.527269] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 858.529728] LNet: Skipped 1 previous similar message [ 863.597698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 863.615538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 863.631123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 866.288509] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:29:17 (1781818157) [ 866.333445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.345058] LNet: 67036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.359173] LNet: Removed LNI 192.168.202.36@tcp [ 867.367643] LNet: Removed LNI 192.168.202.36@tcp1 [ 867.374810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.396532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.405882] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 867.674770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 867.683167] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 867.980390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 868.004964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 868.016851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 868.031432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 868.047533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 868.061347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 868.075637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.36@tcp --health 0 [ 868.112698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 868.447166] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.36@tcp [ 869.471315] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 869.474149] LNet: Skipped 1 previous similar message [ 884.574730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 884.634466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 884.648986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 884.670160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.36@tcp --health 500 [ 884.709171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 891.968573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 891.991325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.000294] LNet: 67554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.023187] LNet: Removed LNI 192.168.202.36@tcp [ 893.030359] LNet: Removed LNI 192.168.202.36@tcp1 [ 893.037329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.060189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.069917] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 893.375827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 893.385313] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 893.707334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 893.723108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 893.749223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 893.764188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 893.781928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 893.804757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 893.821373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 893.839908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.36@tcp --health 0 [ 894.111271] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 894.114347] LNet: Skipped 3 previous similar messages [ 894.957484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 905.389909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 905.403375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 905.418224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 907.902466] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:29:58 (1781818198) [ 908.068856] LNet: 68611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.087132] LNet: Removed LNI 192.168.202.36@tcp [ 909.093728] LNet: Removed LNI 192.168.202.36@tcp1 [ 909.342107] Key type .llcrypt unregistered [ 909.343290] Key type ._llcrypt unregistered [ 911.460099] Key type ._llcrypt registered [ 911.461227] Key type .llcrypt registered [ 911.531430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.538380] alg: No test for adler32 (adler32-zlib) [ 912.347867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 912.360233] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 916.087861] Lustre: DEBUG MARKER: Initial discovery [ 916.115830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.136@tcp [ 917.010152] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 917.059717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 917.085140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.136@tcp [ 917.095677] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 917.105423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 917.121536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.136@tcp -r 1 -m GET -l 3 [ 917.138643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.136@tcp -r 1 -m GET -e local_error [ 917.156307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.136@tcp -r 1 -m PUT -l 6 [ 917.472078] LNet: 1 local NIs in recovery (showing 1): 192.168.202.36@tcp [ 917.717373] Lustre: DEBUG MARKER: Force oleg236-client.virtnet to discover 192.168.202.136@tcp (in background) [ 917.751182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.136@tcp [ 918.265278] Lustre: DEBUG MARKER: Force oleg236-server to discover 192.168.202.36@tcp [ 919.176707] Lustre: DEBUG MARKER: Wait for 69364 [ 920.799218] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 921.278139] Lustre: DEBUG MARKER: Finished wait on 69364 [ 921.310086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 921.327326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 921.419712] LNet: 69821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.469348] LNet: Removed LNI 192.168.202.36@tcp [ 922.607697] Key type .llcrypt unregistered [ 922.609112] Key type ._llcrypt unregistered [ 928.020212] Key type ._llcrypt registered [ 928.021681] Key type .llcrypt registered [ 928.557841] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:30:19 (1781818219) [ 928.731404] Key type .llcrypt unregistered [ 928.733043] Key type ._llcrypt unregistered [ 930.897652] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 930.955268] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 930.957728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 930.959719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 933.150619] Key type ._llcrypt registered [ 933.151856] Key type .llcrypt registered [ 933.217423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 933.223500] alg: No test for adler32 (adler32-zlib) [ 934.025682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.047343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.057209] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 934.336944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 934.344843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 937.318177] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:30:28 (1781818228) [ 937.382776] LNet: 71895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.386307] LNet: Removed LNI 192.168.202.36@tcp [ 937.393514] LNet: Removed LNI 10.1.2.3@tcp [ 937.618341] Key type .llcrypt unregistered [ 937.619813] Key type ._llcrypt unregistered [ 939.853131] Key type ._llcrypt registered [ 939.854764] Key type .llcrypt registered [ 939.920666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.925765] alg: No test for adler32 (adler32-zlib) [ 940.738977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.763578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.774369] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 941.057149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 941.065365] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 941.432675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.36@tcp [ 942.495215] LNetError: Unexpected error -22 connecting to 192.168.202.36@tcp at host 192.168.202.36:988 [ 943.855787] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:30:34 (1781818234) [ 943.893047] LNet: 73018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.927146] LNet: Removed LNI 192.168.202.36@tcp [ 944.934478] LNet: Removed LNI 10.1.2.3@tcp [ 945.188458] Key type .llcrypt unregistered [ 945.190067] Key type ._llcrypt unregistered [ 947.414530] Key type ._llcrypt registered [ 947.416039] Key type .llcrypt registered [ 947.485366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.491705] alg: No test for adler32 (adler32-zlib) [ 948.306745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.333533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 948.345284] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 948.673624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 948.684466] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 949.023079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.36@tcp1 --nid 192.168.202.36@tcp2 [ 950.594174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp1 192.168.202.36@tcp2 [ 950.601264] LNetError: 73860:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.36@tcp1 to 192.168.202.36@tcp2 - Route table may be misconfigured [ 950.612274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp2 192.168.202.36@tcp1 [ 952.845407] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:30:43 (1781818243) [ 952.884812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.901676] LNet: 74427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.951175] LNet: Removed LNI 192.168.202.36@tcp1 [ 954.981450] LNet: Removed LNI 192.168.202.36@tcp2 [ 954.989774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.009973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.019844] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 955.310762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 955.320227] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 955.670133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 955.688441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp 192.168.202.36@tcp [ 955.706722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp1 192.168.202.36@tcp1 [ 955.724281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.36@tcp -d 192.168.202.36@tcp -e network_timeout [ 955.742787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -e network_timeout [ 955.760900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -e network_timeout [ 955.781349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -e network_timeout [ 955.801979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 955.812413] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 955.824788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 955.870187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 955.910479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 955.999110] LNet: 1 local NIs in recovery (showing 1): 192.168.202.36@tcp [ 958.201812] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:30:49 (1781818249) [ 958.242508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.253351] LNet: 75293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.263164] LNet: Removed LNI 192.168.202.36@tcp [ 960.293701] LNet: Removed LNI 192.168.202.36@tcp1 [ 960.302365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.335420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 960.422635] LNet: 75332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.560515] Key type .llcrypt unregistered [ 960.561905] Key type ._llcrypt unregistered [ 964.396420] Key type ._llcrypt registered [ 964.397637] Key type .llcrypt registered [ 964.849115] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:30:55 (1781818255) [ 964.965448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.970494] alg: No test for adler32 (adler32-zlib) [ 965.773119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.235471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 966.755077] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:30:57 (1781818257) [ 966.793704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.802708] LNet: 76426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.813323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.836477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 966.845910] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 967.077731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 967.083954] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 967.394913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 967.405942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp1 [ 967.419553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp1 [ 969.367765] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:31:00 (1781818260) [ 969.920702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg236-server [ 970.451098] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:31:01 (1781818261) [ 971.058538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg236-server [ 971.626451] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:31:02 (1781818262) [ 972.209075] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg236-server [ 972.779959] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:31:03 (1781818263) [ 973.361528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg236-server [ 973.936091] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:31:04 (1781818264) [ 974.560497] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg236-server [ 975.093590] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:31:06 (1781818266) [ 975.672130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg236-server [ 976.221765] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:31:07 (1781818267) [ 976.860924] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg236-server [ 977.454673] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:31:08 (1781818268) [ 978.184952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg236-server [ 978.782355] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:31:09 (1781818269) [ 979.470559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg236-server [ 980.038097] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:31:11 (1781818271) [ 980.079666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.089297] LNet: 80300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.151150] LNet: Removed LNI 192.168.202.36@tcp [ 982.181453] LNet: Removed LNI 192.168.202.36@tcp1 [ 982.189099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.208454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 982.218356] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 982.486771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 982.519226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 982.578068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.587286] LNet: 80450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.653293] LNet: Removed LNI 192.168.202.36@tcp [ 983.661978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.684894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 983.694937] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 983.972997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 984.033204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.059258] LNet: Removed LNI 192.168.202.36@tcp [ 985.067898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.098544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.112374] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 985.381219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 985.413437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 985.481955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.489598] LNet: 80744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.491918] LNet: 80744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 986.533378] LNet: Removed LNI 192.168.202.36@tcp [ 986.542403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.564552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 986.575638] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 986.843782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 986.912848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.942288] LNet: Removed LNI 192.168.202.36@tcp [ 987.950356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.971837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 987.979774] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 988.215151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 988.243129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 988.305139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.349343] LNet: Removed LNI 192.168.202.36@tcp [ 989.357322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.378909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 989.390681] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 989.655548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 989.735358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.744270] LNet: 81181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.747532] LNet: 81181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 990.761334] LNet: Removed LNI 192.168.202.36@tcp [ 990.774408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.798700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 990.810585] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 991.088755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 991.124613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 991.209934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.233109] LNet: Removed LNI 192.168.202.36@tcp [ 992.241996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.274669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 992.295179] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 992.602053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 993.746326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.790586] LNet: Removed LNI 192.168.202.36@tcp [ 994.803204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.826941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 994.838169] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 995.121734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 995.155744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 995.254218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 996.329601] LNet: Removed LNI 192.168.202.36@tcp [ 996.339286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 996.360144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 996.371155] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 996.653678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 997.783432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.792623] LNet: 81785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.795633] LNet: 81785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 998.822068] LNet: Removed LNI 192.168.202.36@tcp [ 998.831915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.873595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.886796] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 999.316689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 999.360720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 999.492522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.549624] LNet: Removed LNI 192.168.202.36@tcp [ 1000.568287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.627681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1000.644977] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1001.185981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 1001.357315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.421169] LNet: Removed LNI 192.168.202.36@tcp [ 1002.445059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.479950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1002.511786] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1003.023559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1003.205457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.235330] LNet: Removed LNI 192.168.202.36@tcp [ 1003.259372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.289759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1003.302116] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1003.829155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1008.606877] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:31:39 (1781818299) [ 1008.723349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.766823] LNet: Removed LNI 192.168.202.36@tcp [ 1008.789470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.821151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1008.835609] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/180] [ 1009.702460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.790713] LNet: Removed LNI 192.168.202.36@tcp231 [ 1009.804300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.848698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1009.867497] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/180] [ 1009.870491] LNet: 83061:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1010.199761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1010.221029] LNet: Removed LNI 192.168.202.36@tcp231 [ 1010.231968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1010.244994] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/180] [ 1010.918508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.944645] LNet: Removed LNI 192.168.202.36@tcp231 [ 1010.961853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.004680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1011.021277] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/0] [ 1011.025860] LNet: 83214:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1011.251338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1011.290619] LNet: Removed LNI 192.168.202.36@tcp231 [ 1011.304164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1011.382501] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/0] [ 1012.065241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.093304] LNet: Removed LNI 192.168.202.36@tcp231 [ 1012.106849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.153364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1012.168635] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/60] [ 1012.171970] LNet: 83366:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1012.410562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1012.437647] LNet: Removed LNI 192.168.202.36@tcp231 [ 1012.449625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1012.469996] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/60] [ 1013.270655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.299533] LNet: Removed LNI 192.168.202.36@tcp231 [ 1013.312338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.358874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1013.377349] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/180] [ 1013.601380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1013.626156] LNet: Removed LNI 192.168.202.36@tcp231 [ 1013.635667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1013.654220] LNet: Added LNI 192.168.202.36@tcp231 [8/256/0/180] [ 1018.763874] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:31:49 (1781818309) [ 1018.909412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.940856] LNet: 84230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.944850] LNet: 84230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1018.961176] LNet: Removed LNI 192.168.202.36@tcp231 [ 1018.973291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.016122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1019.031491] LNet: Added LNI 192.168.202.36@tcp232 [8/256/0/180] [ 1019.552691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1019.577398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1019.598707] LNet: Removed LNI 192.168.202.36@tcp232 [ 1019.614473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1019.642680] LNet: Added LNI 192.168.202.36@tcp232 [8/256/0/180] [ 1020.120723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1020.146297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1020.171879] LNet: Removed LNI 192.168.202.36@tcp232 [ 1020.184203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1020.202767] LNet: Added LNI 192.168.202.36@tcp232 [8/256/0/180] [ 1020.714940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1020.747176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1020.770685] LNet: Removed LNI 192.168.202.36@tcp232 [ 1020.781961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1020.798675] LNet: Added LNI 192.168.202.36@tcp232 [8/256/0/180] [ 1021.299292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1021.322511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1021.345807] LNet: Removed LNI 192.168.202.36@tcp232 [ 1021.356224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1025.868901] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:31:56 (1781818316) [ 1026.433168] Key type .llcrypt unregistered [ 1026.434525] Key type ._llcrypt unregistered [ 1030.572131] Key type ._llcrypt registered [ 1030.573492] Key type .llcrypt registered [ 1030.772906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1030.784228] alg: No test for adler32 (adler32-zlib) [ 1031.661291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1031.684782] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1036.572837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 1040.611221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.629993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1040.643143] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 1041.417547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_interrupt -r 2 [ 1041.439749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_interrupt -r 2 [ 1041.459397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_interrupt -r 2 [ 1041.474694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_interrupt -r 2 [ 1041.493578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_interrupt -r 2 [ 1041.512110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_interrupt -r 2 [ 1041.530723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_interrupt -r 2 [ 1041.557268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_interrupt -r 2 [ 1041.578411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp 192.168.202.136@tcp [ 1041.598867] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 1041.602156] LNet: 1 local NIs in recovery (showing 1): 192.168.202.36@tcp [ 1041.764260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.788491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1041.806744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1041.821872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1041.834920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1041.847993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1041.862848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1041.875645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1041.883844] LNet: 86021:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1041.893467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.921463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.937608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.953235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.973437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.995963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1042.287557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_dropped -r 2 [ 1042.302318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_dropped -r 2 [ 1042.321024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_dropped -r 2 [ 1042.338956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_dropped -r 2 [ 1042.360303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_dropped -r 2 [ 1042.380176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_dropped -r 2 [ 1042.406781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_dropped -r 2 [ 1042.423675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_dropped -r 2 [ 1042.441540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp 192.168.202.136@tcp [ 1042.453137] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 1042.602303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.620504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1042.638332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.655847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.671305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.684767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.697960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.712696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.720814] LNet: 86142:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.728399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.740708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.755848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1042.777462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1042.798909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.816349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.110190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_aborted -r 2 [ 1043.129519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_aborted -r 2 [ 1043.146367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_aborted -r 2 [ 1043.168499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_aborted -r 2 [ 1043.189519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_aborted -r 2 [ 1043.214531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_aborted -r 2 [ 1043.231725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_aborted -r 2 [ 1043.258672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_aborted -r 2 [ 1043.279817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp 192.168.202.136@tcp [ 1043.426318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.453696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1043.478745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.504149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.525269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.541912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.557414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.581697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.600799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.622645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.642836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.658903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.675523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.692180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.971642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_no_route -r 2 [ 1043.992203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_no_route -r 2 [ 1044.013284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_no_route -r 2 [ 1044.034126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_no_route -r 2 [ 1044.063862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_no_route -r 2 [ 1044.089053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_no_route -r 2 [ 1044.116920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_no_route -r 2 [ 1044.140291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_no_route -r 2 [ 1044.156354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp 192.168.202.136@tcp [ 1044.166673] LNet: There was an unexpected network error while writing to 192.168.202.136: rc = -22 [ 1044.168440] LNet: Skipped 2 previous similar messages [ 1044.304476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.325660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1044.351414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.371251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.387973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.407607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.426749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.445751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.458313] LNet: 86382:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.463342] LNet: 86382:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1044.473313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.491430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.506379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.523221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.543951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.560713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.864281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e local_timeout -r 2 [ 1044.884359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e local_timeout -r 2 [ 1044.909614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e local_timeout -r 2 [ 1044.930579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e local_timeout -r 2 [ 1044.950621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e local_timeout -r 2 [ 1044.979954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e local_timeout -r 2 [ 1045.000916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e local_timeout -r 2 [ 1045.030819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e local_timeout -r 2 [ 1045.055439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp 192.168.202.136@tcp [ 1045.196893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.211365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1045.228813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.246786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.264313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1045.278969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1045.295397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.321764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.340502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1045.363866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1045.378739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1045.390596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.409265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1045.430061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.739738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp -m GET -e remote_dropped -r 2 [ 1045.762171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.136@tcp1 -m GET -e remote_dropped -r 2 [ 1045.786755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp -m GET -e remote_dropped -r 2 [ 1045.807239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp -d 192.168.202.36@tcp1 -m GET -e remote_dropped -r 2 [ 1045.836038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp -m GET -e remote_dropped -r 2 [ 1045.859779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.136@tcp1 -m GET -e remote_dropped -r 2 [ 1045.880325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp -m GET -e remote_dropped -r 2 [ 1045.902448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.36@tcp1 -d 192.168.202.36@tcp1 -m GET -e remote_dropped -r 2 [ 1045.924673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.36@tcp 192.168.202.136@tcp [ 1045.943576] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.136@tcp [ 1046.075194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.100909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1046.118902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.131252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.146904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.165827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.180469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.203456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.219649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.244835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1046.271990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.298313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1046.321900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.348286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.118205] LNet: 86696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.144418] LNet: Removed LNI 192.168.202.36@tcp [ 1050.152382] LNet: Removed LNI 192.168.202.36@tcp1 [ 1050.332285] Key type .llcrypt unregistered [ 1050.334273] Key type ._llcrypt unregistered [ 1056.656553] Key type ._llcrypt registered [ 1056.657824] Key type .llcrypt registered [ 1057.228191] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:32:28 (1781818348) [ 1057.409377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.416325] alg: No test for adler32 (adler32-zlib) [ 1058.234312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.264297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1058.279229] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1058.669295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1058.689809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.36@tcp [ 1058.703383] LNetError: 87673:0:(api-ni.c:4295:lnet_ni_set_healthv()) manually adding local NI 192.168.202.36@tcp to recovery [ 1058.749050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.36@tcp [ 1058.798715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.419875] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:32:32 (1781818352) [ 1061.489811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.505084] LNet: 88258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.513862] LNet: Removed LNI 192.168.202.36@tcp [ 1061.528969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.563800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1061.577709] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1061.979951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1061.991603] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 1062.428178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.36@tcp [ 1062.438578] LNetError: 88527:0:(api-ni.c:4295:lnet_ni_set_healthv()) manually adding local NI 192.168.202.36@tcp to recovery [ 1062.559374] LNet: 1 local NIs in recovery (showing 1): 192.168.202.36@tcp [ 1062.560136] LNet: There was an unexpected network error while writing to 192.168.202.36: rc = -22 [ 1063.534164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.36@tcp [ 1067.100882] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:32:38 (1781818358) [ 1067.896629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg236-server [ 1068.598254] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:32:39 (1781818359) [ 1069.393222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg236-server [ 1070.109288] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:32:41 (1781818361) [ 1070.854244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg236-server [ 1071.475242] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:32:42 (1781818362) [ 1071.538443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.548758] LNet: 90147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.607164] LNet: Removed LNI 192.168.202.36@tcp [ 1072.614433] LNet: Removed LNI 192.168.202.36@tcp1 [ 1072.624332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.646796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1072.660331] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1072.942918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1072.972389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1072.996710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1073.027734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1073.049037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1073.067724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1075.347197] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:32:46 (1781818366) [ 1075.386924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1075.397247] LNet: 90871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.407180] LNet: Removed LNI 192.168.202.36@tcp [ 1075.417444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.445572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1075.456872] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1078.215928] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:32:49 (1781818369) [ 1078.244466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1078.257637] LNet: 91569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.265545] LNet: Removed LNI 192.168.202.36@tcp [ 1078.275227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.300171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1078.308343] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1080.992806] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:32:51 (1781818371) [ 1081.604141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1082.340699] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:32:53 (1781818373) [ 1082.526602] LNet: 92609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.541361] LNet: Removed LNI 192.168.202.36@tcp [ 1082.829564] Key type .llcrypt unregistered [ 1082.833913] Key type ._llcrypt unregistered [ 1085.333541] Key type ._llcrypt registered [ 1085.335513] Key type .llcrypt registered [ 1085.462610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1085.472339] alg: No test for adler32 (adler32-zlib) [ 1086.301216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1086.322443] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1089.823461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 1092.724768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.136@tcp [ 1092.917363] LNet: 93024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.925793] LNet: Removed LNI 192.168.202.36@tcp [ 1093.136553] Key type .llcrypt unregistered [ 1093.138209] Key type ._llcrypt unregistered [ 1098.016859] Key type ._llcrypt registered [ 1098.018268] Key type .llcrypt registered [ 1098.646674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1099.313155] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1100.004917] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:33:10 (1781818390) [ 1100.649360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1101.349876] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:33:12 (1781818392) [ 1102.126597] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg236-server [ 1102.769302] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:33:13 (1781818393) [ 1103.512888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg236-server [ 1104.212724] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:33:15 (1781818395) [ 1104.782446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.801402] alg: No test for adler32 (adler32-zlib) [ 1105.680299] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 1105.766864] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1107.375365] Key type lgssc registered [ 1107.864094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.995274] Key type lgssc unregistered [ 1117.114365] LNet: 96194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.121432] LNet: Removed LNI 192.168.202.36@tcp [ 1117.389628] Key type .llcrypt unregistered [ 1117.390614] Key type ._llcrypt unregistered [ 1123.967929] Key type ._llcrypt registered [ 1123.969236] Key type .llcrypt registered [ 1124.555645] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:33:35 (1781818415) [ 1124.756703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1124.768143] alg: No test for adler32 (adler32-zlib) [ 1125.587643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.615285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1125.635416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1125.653728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1125.675667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1128.182330] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:33:39 (1781818419) [ 1128.279498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1128.290637] LNet: 97606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.300843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.335872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1128.349068] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1128.738326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1128.751971] LNet: Removed LNI 192.168.202.36@tcp [ 1128.765072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.790523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1128.808754] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1129.010979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1129.022182] LNet: 97778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.025848] LNet: 97778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1129.036418] LNet: Removed LNI 192.168.202.36@tcp [ 1129.048277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1129.080611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1129.098856] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1131.725557] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:33:42 (1781818422) [ 1131.757777] LNet: 98361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.764472] LNet: Removed LNI 192.168.202.36@tcp [ 1132.010763] Key type .llcrypt unregistered [ 1132.012078] Key type ._llcrypt unregistered [ 1134.303201] Key type ._llcrypt registered [ 1134.304485] Key type .llcrypt registered [ 1134.379827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1134.388842] alg: No test for adler32 (adler32-zlib) [ 1135.199703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.234023] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1135.235980] LNet: *** cfs_fail_loc=e003, val=100*** [ 1135.593462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 1135.605748] Ping reply received with mapping: [ 1135.608436] found metadata reply NI[192.168.202.36@tcp] [ 1135.610162] 24 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1135.612040] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1135.614167] 5a 5a 5a 5a 5a 5a 5a 5a [ 1135.676759] LNet: 98764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.744538] LNet: Removed LNI 192.168.202.36@tcp [ 1137.056476] Key type .llcrypt unregistered [ 1137.058075] Key type ._llcrypt unregistered [ 1141.654780] Key type ._llcrypt registered [ 1141.655827] Key type .llcrypt registered [ 1142.228686] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:33:53 (1781818433) [ 1142.432344] Key type .llcrypt unregistered [ 1142.434289] Key type ._llcrypt unregistered [ 1144.873826] Key type ._llcrypt registered [ 1144.875599] Key type .llcrypt registered [ 1144.966037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.971756] alg: No test for adler32 (adler32-zlib) [ 1145.874399] LNetError: 99824:0:(api-ni.c:2708:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1146.041611] Key type .llcrypt unregistered [ 1146.043732] Key type ._llcrypt unregistered [ 1148.309299] Key type ._llcrypt registered [ 1148.310394] Key type .llcrypt registered [ 1148.868058] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:33:59 (1781818439) [ 1149.113610] Key type .llcrypt unregistered [ 1149.115139] Key type ._llcrypt unregistered [ 1151.667865] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1151.671630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1153.889892] Key type ._llcrypt registered [ 1153.891167] Key type .llcrypt registered [ 1153.961365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1153.968485] alg: No test for adler32 (adler32-zlib) [ 1154.778065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1154.805852] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1157.295429] Lustre: DEBUG MARKER: sanity-lnet test_290: @@@@@@ FAIL: Explicit binding not found for 10.1.2.3: [ 1161.392677] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:34:12 (1781818452) [ 1161.432874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1161.446531] LNet: 101734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.460273] LNet: Removed LNI 10.1.2.3@tcp [ 1161.474548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.795404] Key type .llcrypt unregistered [ 1161.796365] Key type ._llcrypt unregistered [ 1164.143636] Key type ._llcrypt registered [ 1164.145190] Key type .llcrypt registered [ 1164.219018] lnet: unknown parameter 'accept_port_bulk' ignored [ 1164.223750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.230645] alg: No test for adler32 (adler32-zlib) [ 1165.061275] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1165.093395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 1176.021718] Lustre: DEBUG MARKER: sanity-lnet test_291: @@@@@@ FAIL: expected 1 tcp connection on port 989 [ 1180.128138] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:34:31 (1781818471) [ 1180.217607] LNet: 102824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.296292] LNet: Removed LNI 192.168.202.36@tcp [ 1181.609535] Key type .llcrypt unregistered [ 1181.611109] Key type ._llcrypt unregistered [ 1184.111802] Key type ._llcrypt registered [ 1184.112891] Key type .llcrypt registered [ 1184.194815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1184.202059] alg: No test for adler32 (adler32-zlib) [ 1187.966460] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:34:38 (1781818478) [ 1188.006253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1188.026804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1188.071277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1188.087829] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1188.419812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1188.437846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1188.455765] LNet: Removed LNI 192.168.202.36@tcp [ 1188.554512] LNet: 103812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.726431] Key type .llcrypt unregistered [ 1188.727817] Key type ._llcrypt unregistered [ 1192.748399] Key type ._llcrypt registered [ 1192.749993] Key type .llcrypt registered [ 1193.226347] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:34:44 (1781818484) [ 1193.381793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1193.388959] alg: No test for adler32 (adler32-zlib) [ 1194.201860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1194.227749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1194.238778] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1194.513811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 1194.533308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.36@tcp [ 1194.544991] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1194.546827] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1194.549412] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1194, 0, 0 [ 1194.553602] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1194.555588] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1194.557653] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1194.559515] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1194, 0, 0 [ 1194.562974] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1194.564968] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1194.567411] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1194.569767] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1194, 0, 1244 [ 1194.574574] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1194.577028] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1194.579170] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1194.580899] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1194, 0, 1244 [ 1194.583421] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1196.963626] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:34:48 (1781818488) [ 1197.121186] LNet: 105355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.181795] LNet: Removed LNI 192.168.202.36@tcp [ 1198.459510] Key type .llcrypt unregistered [ 1198.461486] Key type ._llcrypt unregistered [ 1200.547762] Key type ._llcrypt registered [ 1200.549388] Key type .llcrypt registered [ 1200.637266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1200.644431] alg: No test for adler32 (adler32-zlib) [ 1201.456455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1201.469798] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1204.365643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 1207.003131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.014954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1207.022194] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 1207.407714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1207.415284] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1210.512211] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:35:01 (1781818501) [ 1210.544506] LNet: 106648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1211.551156] LNet: Removed LNI 192.168.202.36@tcp [ 1212.575094] LNet: Removed LNI 192.168.202.36@tcp1 [ 1212.583095] LNet: Removed LNI 10.1.2.3@tcp99 [ 1212.860699] Key type .llcrypt unregistered [ 1212.862722] Key type ._llcrypt unregistered [ 1215.045689] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1215.047667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1217.246671] Key type ._llcrypt registered [ 1217.248168] Key type .llcrypt registered [ 1217.323764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1217.331807] alg: No test for adler32 (adler32-zlib) [ 1218.154799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.179031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1218.191952] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1218.485572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1218.494386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1219.306651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1219.324950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1219.344781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1219.355643] LNetError: 107399:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1219.371462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1219.389185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1219.555485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1219.581709] LNet: 107419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.639122] LNet: Removed LNI 192.168.202.36@tcp [ 1220.646338] LNet: Removed LNI 10.1.2.3@tcp [ 1220.860513] Key type .llcrypt unregistered [ 1220.861713] Key type ._llcrypt unregistered [ 1224.587844] Key type ._llcrypt registered [ 1224.588589] Key type .llcrypt registered [ 1225.033422] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:35:16 (1781818516) [ 1225.151068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1225.157128] alg: No test for adler32 (adler32-zlib) [ 1225.958775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.981553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1225.991314] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1226.254474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.36@tcp [ 1226.281322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg236-client@tcp [ 1228.469984] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:35:19 (1781818519) [ 1228.506037] LNet: 108941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.544070] LNet: Removed LNI 192.168.202.36@tcp [ 1229.772380] Key type .llcrypt unregistered [ 1229.773504] Key type ._llcrypt unregistered [ 1231.806498] Key type ._llcrypt registered [ 1231.807472] Key type .llcrypt registered [ 1231.864048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1231.870405] alg: No test for adler32 (adler32-zlib) [ 1232.670894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1232.682027] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1232.693969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1234.702194] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:35:25 (1781818525) [ 1235.178673] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1235.711727] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:35:26 (1781818526) [ 1236.159762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1236.684642] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:35:27 (1781818527) [ 1236.738786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1236.748978] LNet: 110365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.754456] LNet: Removed LNI 192.168.202.36@tcp [ 1236.760904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.779703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1236.788386] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1237.067983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1237.092308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1239.361950] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:35:30 (1781818530) [ 1239.389659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1239.399671] LNet: 111069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.421752] LNet: Removed LNI 192.168.202.36@tcp [ 1240.428879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.449773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1240.463432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1240.559604] LNet: 111111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.684408] Key type .llcrypt unregistered [ 1240.685454] Key type ._llcrypt unregistered [ 1244.468538] Key type ._llcrypt registered [ 1244.469779] Key type .llcrypt registered [ 1244.915498] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:35:35 (1781818535) [ 1245.050428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1245.056525] alg: No test for adler32 (adler32-zlib) [ 1245.864537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.884550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1245.893547] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1246.152401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1246.175991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1248.682887] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:35:39 (1781818539) [ 1248.719117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1248.730330] LNet: 112622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.766659] LNet: Removed LNI 192.168.202.36@tcp [ 1249.777173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.804622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1249.823240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1252.065071] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:35:43 (1781818543) [ 1252.102514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1252.114356] LNet: 113203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.126310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.160421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1252.316731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1252.329501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1252.351628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1252.506251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1252.523729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1252.556537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1252.574564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.606963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1252.746572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1252.761288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1252.784355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1252.936621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1252.951285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1255.141557] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:35:46 (1781818546) [ 1255.170434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1255.181023] LNet: 113835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.183372] LNet: 113835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1255.193064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1255.215795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1255.224092] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1255.489621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1255.498026] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1255.806699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1255.830638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1255.913733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1256.927147] LNet: Removed LNI 192.168.202.36@tcp [ 1256.934338] LNet: Removed LNI 192.168.202.36@tcp2 [ 1256.941952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.971188] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1256.974290] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1256.976956] LNet: 114126:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1257.166428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1257.242049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1257.250685] LNet: 114163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.252881] LNet: 114163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1258.271180] LNet: Removed LNI 192.168.202.36@tcp [ 1258.283364] LNet: Removed LNI 192.168.202.36@tcp2 [ 1258.292721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.316289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1258.324933] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1258.580893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1258.589668] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1258.878933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1258.896273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1258.917191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1258.993977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1260.063139] LNet: Removed LNI 192.168.202.36@tcp [ 1261.093560] LNet: Removed LNI 192.168.202.36@tcp2 [ 1261.102861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.127746] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1261.129733] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1261.131633] LNet: 114453:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1261.313701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1261.382735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1261.392206] LNet: 114490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.394669] LNet: 114490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1261.397674] LNet: Removed LNI 192.168.202.36@tcp [ 1262.438292] LNet: Removed LNI 192.168.202.36@tcp2 [ 1262.447104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1262.467860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1262.476745] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1262.735710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1262.743129] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1263.050898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1263.070641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1263.145559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.159121] LNet: Removed LNI 192.168.202.36@tcp [ 1264.170333] LNet: Removed LNI 192.168.202.36@tcp2 [ 1264.179442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1264.205057] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1264.207232] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1264.209306] LNet: 114776:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1264.374091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1264.439224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1265.503128] LNet: Removed LNI 192.168.202.36@tcp [ 1265.509588] LNet: Removed LNI 192.168.202.36@tcp2 [ 1265.517252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1265.542099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1265.553880] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1265.808484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1265.818419] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1266.145911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1266.160821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1266.182834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1266.262133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1267.295120] LNet: Removed LNI 192.168.202.36@tcp [ 1268.325245] LNet: Removed LNI 192.168.202.36@tcp2 [ 1268.332841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1268.355142] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1268.357597] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1268.359505] LNet: 115103:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1268.505596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1268.579082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1268.588581] LNet: Removed LNI 192.168.202.36@tcp [ 1269.605286] LNet: Removed LNI 192.168.202.36@tcp2 [ 1269.613369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.632736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1269.643563] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1269.908503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1269.914695] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1270.197059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1270.208429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1270.227955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1270.349015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1270.355947] LNet: 115437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.359221] LNet: 115437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1271.391138] LNet: Removed LNI 192.168.202.36@tcp [ 1271.403299] LNet: Removed LNI 192.168.202.36@tcp2 [ 1271.411388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.434264] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1271.436785] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1271.592414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1271.701936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1272.735156] LNet: Removed LNI 192.168.202.36@tcp [ 1272.743395] LNet: Removed LNI 192.168.202.36@tcp2 [ 1272.751509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.776150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1272.787199] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1273.013604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1273.023021] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1273.322879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1273.336947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1273.349157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1273.369838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1273.485304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1274.527133] LNet: Removed LNI 192.168.202.36@tcp [ 1275.557459] LNet: Removed LNI 192.168.202.36@tcp2 [ 1275.566102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.590710] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1275.593386] LNet: Added LNI 192.168.202.36@tcp2 [8/256/0/180] [ 1275.595318] LNet: 115802:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.597577] LNet: 115802:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1275.749421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1278.183185] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1278.773195] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:36:09 (1781818569) [ 1278.802248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1278.814629] LNet: Removed LNI 192.168.202.36@tcp [ 1278.826725] LNet: Removed LNI 192.168.202.36@tcp2 [ 1278.836128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.942829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1278.953923] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1279.247495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1279.256507] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1279.595558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1279.606855] LNet: Removed LNI 192.168.202.36@tcp [ 1279.613354] LNet: Removed LNI 10.1.2.3@tcp [ 1279.622250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1279.652707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1279.664262] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1279.666729] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1279.817951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1279.828735] LNet: Removed LNI 192.168.202.36@tcp [ 1279.842412] LNet: Removed LNI 10.1.2.3@tcp [ 1282.024468] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:36:13 (1781818573) [ 1282.060766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1282.077219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1282.114423] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1282.159591] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1282.162119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1282.164270] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1284.298270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1284.308538] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1284.571584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1284.581074] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1285.330875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.36@tcp [ 1285.354798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1285.381908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.36@tcp -m PUT -e local_timeout -r 1 [ 1285.406893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1285.429441] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1285.430915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1285.517825] LNetError: 117496:0:(lib-move.c:4714:lnet_parse()) 192.168.202.36@tcp, src 192.168.202.36@tcp: Dropping PUT (error -108 looking up sender) [ 1286.559532] LNet: 117954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.563861] LNet: 117954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1287.583161] LNet: Removed LNI 192.168.202.36@tcp [ 1287.593484] LNet: Removed LNI 10.1.2.3@tcp [ 1287.741709] Key type .llcrypt unregistered [ 1287.742976] Key type ._llcrypt unregistered [ 1291.605346] Key type ._llcrypt registered [ 1291.606562] Key type .llcrypt registered [ 1292.149411] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:36:23 (1781818583) [ 1292.320912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1292.328713] alg: No test for adler32 (adler32-zlib) [ 1293.162875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1293.376301] LNet: 118807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.638911] Key type .llcrypt unregistered [ 1293.641594] Key type ._llcrypt unregistered [ 1296.257425] Key type ._llcrypt registered [ 1296.258789] Key type .llcrypt registered [ 1296.369632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1296.377559] alg: No test for adler32 (adler32-zlib) [ 1297.229621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1297.247045] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1303.435719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 1303.604375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1304.194920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.136@tcp [ 1304.417252] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.136@tcp [ 1304.477433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.517343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1304.554375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.579827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.613290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.646387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.671650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.711724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.747181] LNet: 119256:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.771908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.837378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.884209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.909831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.936339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.965461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.480479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1305.745243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.781299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1305.817495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.843822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.869752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.894164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.912316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.939348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.970623] LNet: 119373:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.001280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.048622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.082865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.105305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.140273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.165800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.693366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.136@tcp [ 1306.862106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.884336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1306.901984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.924905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.947661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.973461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.992253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.014671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.025869] LNet: 119490:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.038527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.060187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.078808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.099418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.122996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.145304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1307.710466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1308.012334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1308.072755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1308.116484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1308.145157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1308.187685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1308.252978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1308.308775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1308.352208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1308.396275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1308.439242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1308.471675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.549951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1308.604655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.646718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.447303] LNet: 119654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.505266] LNet: Removed LNI 192.168.202.36@tcp [ 1310.840440] Key type .llcrypt unregistered [ 1310.842486] Key type ._llcrypt unregistered [ 1325.610626] Key type ._llcrypt registered [ 1325.613580] Key type .llcrypt registered [ 1326.749358] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:36:57 (1781818617) [ 1327.372913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.389512] alg: No test for adler32 (adler32-zlib) [ 1328.282160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.598478] LNet: 120543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.050888] Key type .llcrypt unregistered [ 1329.053219] Key type ._llcrypt unregistered [ 1335.776868] Key type ._llcrypt registered [ 1335.779916] Key type .llcrypt registered [ 1336.006337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1336.016283] alg: No test for adler32 (adler32-zlib) [ 1336.919347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1336.940690] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1342.769617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.136@tcp [ 1348.635719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.720210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1348.774368] LNet: Added LNI 192.168.202.36@tcp1 [8/256/0/180] [ 1349.449786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1350.097650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.136@tcp [ 1350.239521] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.136@tcp [ 1350.276181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.299318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1350.326752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.350314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.376339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.397948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.417675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.437984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.448364] LNet: 121196:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1350.460111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.479589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1350.498907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.534874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1350.574230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.610595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.129516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1351.256672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1351.280508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1351.298480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1351.320349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1351.339853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1351.363592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1351.382707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1351.401690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1351.413338] LNet: 121313:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1351.424594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1351.443651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1351.470082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.489310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1351.510337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.536725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.910723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.136@tcp [ 1352.045301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1352.061675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1352.078489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1352.096431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1352.115393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1352.133777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1352.154288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1352.174361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1352.194187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1352.216822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1352.248511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1352.280690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1352.307294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1352.333052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1352.763114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1352.996191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1353.019482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1353.047896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1353.075873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1353.100306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1353.127990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1353.160328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1353.188563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1353.212903] LNet: 121547:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1353.220502] LNet: 121547:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1353.239187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1353.263296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1353.283724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1353.307844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.348679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1353.384441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.707094] LNet: 121629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.759210] LNet: Removed LNI 192.168.202.36@tcp [ 1361.800098] LNet: Removed LNI 192.168.202.36@tcp1 [ 1362.779890] Key type .llcrypt unregistered [ 1362.782407] Key type ._llcrypt unregistered [ 1388.900777] Key type ._llcrypt registered [ 1388.905280] Key type .llcrypt registered [ 1391.969864] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:38:00 (1781818680) [ 1410.528261] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:38:20 (1781818700) [ 1423.388880] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:38:32 (1781818712) [ 1432.712585] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:38:42 (1781818722) [ 1441.073764] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:38:50 (1781818730) [ 1448.469698] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:38:58 (1781818738) [ 1456.822567] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:39:06 (1781818746) [ 1465.935239] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:39:15 (1781818755) [ 1474.782332] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:39:24 (1781818764) [ 1483.219277] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:39:33 (1781818773) [ 1489.935604] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:39:40 (1781818780) [ 1497.722360] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:39:47 (1781818787) [ 1504.986202] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:39:55 (1781818795) [ 1511.298063] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:40:01 (1781818801) [ 1519.095925] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:40:09 (1781818809) [ 1528.145882] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:40:17 (1781818817) [ 1537.235532] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:40:27 (1781818827) [ 1546.075721] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:40:35 (1781818835) [ 1546.775085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1546.819492] alg: No test for adler32 (adler32-zlib) [ 1547.833923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.013948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1548.032622] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1550.192769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.262456] LNet: 132350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.286238] LNet: Removed LNI 192.168.202.36@tcp [ 1550.345481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.407324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1550.430100] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1560.752774] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:40:50 (1781818850) [ 1569.969980] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:40:59 (1781818859) [ 1579.112738] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:41:09 (1781818869) [ 1585.918298] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:41:15 (1781818875) [ 1595.172664] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:41:24 (1781818884) [ 1603.979374] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:41:33 (1781818893) [ 1611.398472] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:41:41 (1781818901) [ 1619.669638] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:41:49 (1781818909) [ 1628.182490] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:41:58 (1781818918) [ 1637.500541] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:42:07 (1781818927) [ 1646.151521] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:42:16 (1781818936) [ 1654.488095] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:42:24 (1781818944) [ 1662.838673] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1436 sec ========== 17:42:32 (1781818952) [ 1663.503244] LNet: 139820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.535480] LNet: Removed LNI 192.168.202.36@tcp [ 1664.401978] Key type .llcrypt unregistered [ 1664.410673] Key type ._llcrypt unregistered [ 1679.654929] Key type ._llcrypt registered [ 1679.657208] Key type .llcrypt registered [ 1680.528209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1680.566226] alg: No test for adler32 (adler32-zlib) [ 1681.782249] Lustre: Lustre: Build Version: 2.17.54_40_g9c4a26d [ 1682.141047] LNet: Added LNI 192.168.202.36@tcp [8/256/0/180] [ 1683.927347] Key type lgssc registered [ 1685.551524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1753.710900] Lustre: Mounted lustre-client [ 1758.560403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1766.712983] Lustre: Unmounted lustre-client [ 1790.801515] Key type lgssc unregistered [ 1791.006707] LNet: 142748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.037758] LNet: Removed LNI 192.168.202.36@tcp [ 1792.495311] Key type .llcrypt unregistered [ 1792.497753] Key type ._llcrypt unregistered