[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1081071870 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002368] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004022] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010171] LSM: Security Framework initializing [ 0.011074] Yama: becoming mindful. [ 0.012062] SELinux: Initializing. [ 0.013090] *** VALIDATE selinux *** [ 0.030255] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.037623] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.038432] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039166] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040168] *** VALIDATE tmpfs *** [ 0.041545] *** VALIDATE proc *** [ 0.042651] *** VALIDATE cgroup *** [ 0.043025] *** VALIDATE cgroup2 *** [ 0.045128] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.047072] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.048000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.048042] Spectre V2 : User space: Vulnerable [ 0.049023] Speculative Store Bypass: Vulnerable [ 0.054331] debug: unmapping init [mem 0xffffffffb5459000-0xffffffffb5460fff] [ 0.058534] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.061088] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.062048] ... version: 2 [ 0.063023] ... bit width: 48 [ 0.064036] ... generic registers: 4 [ 0.065028] ... value mask: 0000ffffffffffff [ 0.066018] ... max period: 00007fffffffffff [ 0.067026] ... fixed-purpose events: 3 [ 0.068014] ... event mask: 000000070000000f [ 0.069387] rcu: Hierarchical SRCU implementation. [ 0.074000] smp: Bringing up secondary CPUs ... [ 0.074741] x86: Booting SMP configuration: [ 0.075035] .... node #0, CPUs: #1 #2 #3 [ 0.086479] smp: Brought up 1 node, 4 CPUs [ 0.088043] smpboot: Max logical packages: 1 [ 0.089018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.476036] node 0 deferred pages initialised in 385ms [ 0.483133] devtmpfs: initialized [ 0.486503] x86/mm: Memory block size: 128MB [ 0.492035] gcov: version magic: 0x41383552 [ 0.496577] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.501117] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.505595] pinctrl core: initialized pinctrl subsystem [ 0.507509] [ 0.508011] ************************************************************* [ 0.511018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.514077] ** ** [ 0.518032] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.521017] ** ** [ 0.523016] ** This means that this kernel is built to expose internal ** [ 0.527020] ** IOMMU data structures, which may compromise security on ** [ 0.530219] ** your system. ** [ 0.533019] ** ** [ 0.537018] ** If you see this message and you are not debugging the ** [ 0.540018] ** kernel, report this immediately to your vendor! ** [ 0.543020] ** ** [ 0.546020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.548015] ************************************************************* [ 0.552233] NET: Registered protocol family 16 [ 0.554832] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.557093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.562348] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.567083] cpuidle: using governor menu [ 0.570255] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.575087] PCI: Using configuration type 1 for base access [ 0.578180] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.597602] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.601069] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.609048] cryptd: max_cpu_qlen set to 1000 [ 0.616688] ACPI: Added _OSI(Module Device) [ 0.620027] ACPI: Added _OSI(Processor Device) [ 0.623027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.626025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.639209] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.654137] ACPI: Interpreter enabled [ 0.655096] ACPI: PM: (supports S0 S3 S4 S5) [ 0.658022] ACPI: Using IOAPIC for interrupt routing [ 0.660140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.666488] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.690038] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.692060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.695028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.716345] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.728897] acpiphp: Slot [2] registered [ 0.729000] acpiphp: Slot [5] registered [ 0.729172] acpiphp: Slot [6] registered [ 0.732275] acpiphp: Slot [7] registered [ 0.737203] acpiphp: Slot [8] registered [ 0.751239] acpiphp: Slot [9] registered [ 0.753206] acpiphp: Slot [10] registered [ 0.759211] acpiphp: Slot [3] registered [ 0.760135] acpiphp: Slot [4] registered [ 0.932866] acpiphp: Slot [11] registered [ 0.937081] acpiphp: Slot [12] registered [ 0.938582] acpiphp: Slot [13] registered [ 0.943213] acpiphp: Slot [14] registered [ 0.945187] acpiphp: Slot [15] registered [ 0.948177] acpiphp: Slot [16] registered [ 0.949230] acpiphp: Slot [17] registered [ 0.950000] acpiphp: Slot [18] registered [ 0.950000] acpiphp: Slot [19] registered [ 0.950000] acpiphp: Slot [20] registered [ 0.950000] acpiphp: Slot [21] registered [ 0.950000] acpiphp: Slot [22] registered [ 0.950000] acpiphp: Slot [23] registered [ 0.950000] acpiphp: Slot [24] registered [ 0.950000] acpiphp: Slot [25] registered [ 0.950000] acpiphp: Slot [26] registered [ 0.950000] acpiphp: Slot [27] registered [ 0.950000] acpiphp: Slot [28] registered [ 0.950000] acpiphp: Slot [29] registered [ 0.950000] acpiphp: Slot [30] registered [ 0.950000] acpiphp: Slot [31] registered [ 0.950000] PCI host bridge to bus 0000:00 [ 0.950000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.955208] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.958027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.961029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.964031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.968046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.971243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.976992] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.982098] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.997821] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.005869] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.010032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.013026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.018032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.024561] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.029653] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.034484] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.040207] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.051024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.070023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 1.079019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.088197] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.100024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.112022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.145030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 1.161918] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.177034] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.189038] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.208034] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.231000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.239024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.242000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.255022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.267000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.276027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.286040] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.307021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.323321] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.330024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.338022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.365023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.405034] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.415023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.422026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.430000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.430000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.435467] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.440015] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.445621] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.452043] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.463056] iommu: Default domain type: Passthrough [ 1.468965] SCSI subsystem initialized [ 1.471179] ACPI: bus type USB registered [ 1.473176] usbcore: registered new interface driver usbfs [ 1.478140] usbcore: registered new interface driver hub [ 1.483156] usbcore: registered new device driver usb [ 1.486658] pps_core: LinuxPPS API ver. 1 registered [ 1.490017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.497101] PTP clock support registered [ 1.502367] EDAC MC: Ver: 3.0.0 [ 1.505117] PCI: Using ACPI for IRQ routing [ 1.507808] NetLabel: Initializing [ 1.508015] NetLabel: domain hash size = 128 [ 1.509015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.510112] NetLabel: unlabeled traffic allowed by default [ 1.511757] vgaarb: loaded [ 1.512515] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.514012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.521417] clocksource: Switched to clocksource kvm-clock [ 1.793136] VFS: Disk quotas dquot_6.6.0 [ 1.795884] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.800250] *** VALIDATE ramfs *** [ 1.802691] *** VALIDATE hugetlbfs *** [ 1.805592] pnp: PnP ACPI init [ 1.811118] pnp: PnP ACPI: found 6 devices [ 1.838667] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.843358] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.848564] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.850945] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.855513] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.861600] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.866923] NET: Registered protocol family 2 [ 1.873193] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.884985] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.893845] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.910596] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.920896] TCP: Hash tables configured (established 65536 bind 65536) [ 1.926973] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.934532] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.941894] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.950807] NET: Registered protocol family 1 [ 1.960918] RPC: Registered named UNIX socket transport module. [ 1.968674] RPC: Registered udp transport module. [ 1.973528] RPC: Registered tcp transport module. [ 2.044062] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.051884] NET: Registered protocol family 44 [ 2.057017] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.065646] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.074643] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.082867] PCI: CLS 0 bytes, default 64 [ 2.090023] Unpacking initramfs... [ 7.599629] debug: unmapping init [mem 0xffff8a37fcc54000-0xffff8a37fffbffff] [ 7.618018] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 7.622542] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 7.634131] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 8.586891] Initialise system trusted keyrings [ 8.589300] Key type blacklist registered [ 8.591723] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 8.605861] zbud: loaded [ 8.625231] *** VALIDATE nfs *** [ 8.628689] *** VALIDATE nfs4 *** [ 8.640571] pstore: using deflate compression [ 8.673530] Platform Keyring initialized [ 9.317870] NET: Registered protocol family 38 [ 9.324082] Key type asymmetric registered [ 9.329950] Asymmetric key parser 'x509' registered [ 9.342292] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 9.362671] io scheduler mq-deadline registered [ 9.370381] io scheduler kyber registered [ 9.372215] io scheduler bfq registered [ 9.379846] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 9.394519] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 9.407903] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 9.416055] ACPI: Power Button [PWRF] [ 9.433149] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 9.456952] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.512704] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.548247] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.584569] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.637518] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.697634] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.714973] Non-volatile memory driver v1.3 [ 9.725025] Linux agpgart interface v0.103 [ 9.910078] virtio_blk virtio1: [vda] 135128 512-byte logical blocks (69.2 MB/66.0 MiB) [ 9.928501] vda: detected capacity change from 0 to 69185536 [ 9.992693] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.998697] vdb: detected capacity change from 0 to 1073741824 [ 10.088815] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 10.109114] vdc: detected capacity change from 0 to 2621440000 [ 10.202731] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 10.221565] vdd: detected capacity change from 0 to 2621440000 [ 10.289721] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.297970] vde: detected capacity change from 0 to 4294967296 [ 10.351992] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.358616] vdf: detected capacity change from 0 to 4294967296 [ 10.372406] libphy: Fixed MDIO Bus: probed [ 10.432418] usbcore: registered new interface driver usbserial_generic [ 10.444805] usbserial: USB Serial support registered for generic [ 10.449367] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 10.462667] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 10.464919] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 10.477676] mousedev: PS/2 mouse device common for all mice [ 10.484943] rtc_cmos 00:05: RTC can wake from S4 [ 10.491927] rtc_cmos 00:05: registered as rtc0 [ 10.496028] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 10.502643] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 10.511894] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 10.512437] intel_pstate: CPU model not supported [ 10.528567] hid: raw HID events driver (C) Jiri Kosina [ 10.542800] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 10.555972] usbcore: registered new interface driver usbhid [ 10.575189] usbhid: USB HID core driver [ 10.583496] drop_monitor: Initializing network drop monitor service [ 10.591260] Initializing XFRM netlink socket [ 10.599869] NET: Registered protocol family 10 [ 10.607361] Segment Routing with IPv6 [ 10.610811] NET: Registered protocol family 17 [ 10.621225] mpls_gso: MPLS GSO support [ 10.637829] RAS: Correctable Errors collector initialized. [ 10.644100] AVX version of gcm_enc/dec engaged. [ 10.655623] AES CTR mode by8 optimization enabled [ 10.934731] sched_clock: Marking stable (10934708621, 0)->(13960825397, -3026116776) [ 10.950638] registered taskstats version 1 [ 10.955310] Loading compiled-in X.509 certificates [ 10.964465] zswap: loaded using pool lzo/zbud [ 11.039104] Key type big_key registered [ 11.083518] Key type encrypted registered [ 11.087264] ima: No TPM chip found, activating TPM-bypass! [ 11.095536] ima: Allocated hash algorithm: sha1 [ 11.103798] ima: No architecture policies found [ 11.106218] evm: Initialising EVM extended attributes: [ 11.116100] evm: security.selinux [ 11.120554] evm: security.ima [ 11.123985] evm: security.capability [ 11.128582] evm: HMAC attrs: 0x1 [ 11.145470] rtc_cmos 00:05: setting system clock to 2026-03-10 00:17:31 UTC (1773101851) [ 11.166762] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 11.178129] debug: unmapping init [mem 0xffffffffb5182000-0xffffffffb5458fff] [ 11.192760] Write protecting the kernel read-only data: 28672k [ 11.207431] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 11.221900] debug: unmapping init [mem 0xffffffffb4114000-0xffffffffb41fffff] [ 11.421596] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.462704] systemd[1]: Detected virtualization kvm. [ 11.477295] systemd[1]: Detected architecture x86-64. [ 11.485386] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 11.584581] systemd[1]: No hostname configured. [ 11.594512] systemd[1]: Set hostname to . [ 11.599480] random: systemd: uninitialized urandom read (16 bytes read) [ 11.609407] systemd[1]: Initializing machine ID from random generator. [ 12.027605] random: systemd: uninitialized urandom read (16 bytes read) [ 12.030783] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 12.039941] random: systemd: uninitialized urandom read (16 bytes read) [ 12.045540] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 12.056575] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service.[ 12.989823] random: fast init done [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 14.090510] device-mapper: uevent: version 1.0.3 [ 14.096229] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 16.470154] virtio_net virtio0 ens2: renamed from eth0 [ 17.061302] scsi host0: ata_piix [ 17.155651] scsi host1: ata_piix [ 17.173415] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 17.193459] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 22.703500] random: crng init done [ 22.711146] random: 7 urandom warning(s) missed due to ratelimiting [ 25.162359] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 27.571865] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 30.625792] printk: systemd: 26 output lines suppressed due to ratelimiting [ 32.011612] SELinux: Disabled at runtime. [ 32.168529] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.197382] systemd[1]: Detected virtualization kvm. [ 32.201065] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 34.323576] systemd[1]: initrd-switch-root.service: Succeeded. [ 34.333108] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 34.340803] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 34.354307] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 34.365806] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 34.384497] systemd[1]: Starting Journal Service... Starting Journal Service... [ 34.405951] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ 34.760704] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 35.888925] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.219685] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.348843] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 38.006539] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.159769] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 44.251327] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit)[ 45.735557] NFS: Registering the id_resolver key type [ 45.738976] Key type id_resolver registered [ 45.742950] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg135-server login: [ 100.707414] libcfs: loading out-of-tree module taints kernel. [ 100.749415] Key type ._llcrypt registered [ 100.751330] Key type .llcrypt registered [ 100.829823] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 115.277384] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 116.371581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.384449] alg: No test for adler32 (adler32-zlib) [ 117.717919] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 118.498987] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 120.191525] Key type lgssc registered [ 121.697567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.141696] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.293050] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 183.439932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.480163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.761121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.790752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.874686] Lustre: lustre-MDT0000: new disk, initializing [ 184.998390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.018813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.141577] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.634408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.717627] Lustre: 6495:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.766304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.769841] Lustre: Skipped 1 previous similar message [ 198.848868] Lustre: lustre-MDT0001: new disk, initializing [ 198.891063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.911637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.917975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.415398] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.189986] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.736599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.905552] Lustre: lustre-OST0000: new disk, initializing [ 213.910353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 213.967861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.024497] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.162294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.175906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.239107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.282973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.371358] Lustre: lustre-OST0001: new disk, initializing [ 231.378901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 231.431027] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.135841] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.601970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.614472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.697883] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 246.134951] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.635429] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 260.320878] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.397710] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing yml_node [ 267.685297] Lustre: DEBUG MARKER: Client: 2.17.50.236 [ 269.748627] Lustre: DEBUG MARKER: MDS: 2.17.50.236 [ 271.767153] Lustre: DEBUG MARKER: OSS: 2.17.50.236 [ 273.106689] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 9 20:21:51 EDT 2026 [ 287.874860] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 288.973693] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 304.096634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.103082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.117890] Lustre: Skipped 3 previous similar messages [ 304.124110] Lustre: Skipped 2 previous similar messages [ 309.221770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.226468] Lustre: Skipped 4 previous similar messages [ 309.436194] Lustre: server umount lustre-MDT0000 complete [ 312.365219] LustreError: 9453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102152 with bad export cookie 10205460633341348911 [ 312.370426] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.375499] LustreError: 9453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 314.342072] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.349693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 314.358413] Lustre: Skipped 1 previous similar message [ 317.410110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.415445] Lustre: Skipped 2 previous similar messages [ 318.650883] Lustre: server umount lustre-MDT0001 complete [ 321.922676] Lustre: server umount lustre-OST0000 complete [ 324.891414] Lustre: server umount lustre-OST0001 complete [ 329.588123] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 336.633733] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 370.394465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.574733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 370.602596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 370.673714] Lustre: lustre-MDT0000: new disk, initializing [ 370.747031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.766802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 374.183480] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 383.545434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.632822] Lustre: 14970:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 383.695798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 383.699809] Lustre: Skipped 1 previous similar message [ 383.763870] Lustre: lustre-MDT0001: new disk, initializing [ 383.846588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 383.882437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 383.896501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 387.859541] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 394.263564] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 398.087396] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 403.834450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 404.100385] Lustre: lustre-OST0000: new disk, initializing [ 404.105493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 404.228655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.350283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 405.361585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 405.426236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 407.011969] hrtimer: interrupt took 7171868 ns [ 409.202927] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 415.772486] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 419.538844] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 419.721420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 423.194102] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 423.406383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 425.955963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 425.961693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.964574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 425.974530] Lustre: Skipped 1 previous similar message [ 425.983706] Lustre: Skipped 1 previous similar message [ 436.194314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.201192] Lustre: Skipped 3 previous similar messages [ 439.263140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 439.385910] Lustre: server umount lustre-OST0000 complete [ 445.410634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 445.417871] LustreError: Skipped 1 previous similar message [ 445.420549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.756407] Lustre: server umount lustre-MDT0000 complete [ 451.554363] LustreError: 14981:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.571700] LustreError: 14981:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 451.960846] LustreError: 14961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102292 with bad export cookie 10205460633341350444 [ 451.967546] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 451.974255] LustreError: 14961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 452.329877] Lustre: server umount lustre-MDT0001 complete [ 457.502560] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:24:56 (1773102296) [ 472.699475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.142564] LustreError: 19439:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.198721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.254174] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 478.690914] LustreError: 19440:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.760259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.785619] LustreError: 19439:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.262810] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 490.565303] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 493.453202] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 495.299193] Lustre: Failing over lustre-MDT0000 [ 497.547020] Lustre: server umount lustre-MDT0000 complete [ 498.657732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.668234] LustreError: 19434:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.674211] Lustre: Skipped 2 previous similar messages [ 498.690526] LustreError: 19434:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 507.877453] LustreError: 19435:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.891184] LustreError: 19435:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 515.039106] Lustre: 3649:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773102339/real 1773102339] req@ffff8a3744bd4700 x1859232158859392/t0(0) o400->MGC192.168.201.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773102355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 515.065402] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.150857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.280654] LustreError: 3647:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a386d5c8380 x1859232158862080/t0(0) o250->MGC192.168.201.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 525.300188] LustreError: 19439:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.320719] LustreError: 19439:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 528.630690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 528.631839] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 528.673623] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 529.407850] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 536.290531] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 539.769974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.778972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.783660] Lustre: Skipped 3 previous similar messages [ 544.224741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 546.028289] Lustre: server umount lustre-MDT0000 complete [ 549.060019] LustreError: 19419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102389 with bad export cookie 10205460633341351858 [ 549.061397] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.070142] LustreError: 19419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 549.379633] Lustre: server umount lustre-MDT0001 complete [ 556.250720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.682529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.688678] Lustre: Skipped 2 previous similar messages [ 560.888421] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 562.150188] LustreError: 22434:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.175235] LustreError: 22434:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 569.708120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.074550] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 578.948706] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 582.482452] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 584.417696] Lustre: Failing over lustre-MDT0000 [ 584.616073] Lustre: server umount lustre-MDT0000 complete [ 585.696827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.712976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.738794] Lustre: Skipped 1 previous similar message [ 602.059981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.145528] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.395114] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 605.566877] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 607.718611] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 607.727392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 607.734790] Lustre: Skipped 1 previous similar message [ 607.749520] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 612.118320] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 617.952689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 617.962098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.970264] Lustre: Skipped 1 previous similar message [ 617.974662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 617.983523] Lustre: Skipped 3 previous similar messages [ 621.809215] Lustre: server umount lustre-MDT0000 complete [ 625.248837] LustreError: 22416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102465 with bad export cookie 10205460633341352796 [ 625.250445] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.259936] LustreError: 22416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 625.551413] Lustre: server umount lustre-MDT0001 complete [ 634.245097] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:27:52 (1773102472) [ 647.435805] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 655.643338] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 686.248575] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 694.692641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.882373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 694.902674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 694.952135] Lustre: lustre-MDT0000: new disk, initializing [ 695.019808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 695.025663] Lustre: Skipped 2 previous similar messages [ 695.036927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 697.672248] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 705.287830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.363345] Lustre: 28413:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 705.557757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 708.480989] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 713.879541] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 716.468766] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 721.608504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 723.515159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 723.571271] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 723.649381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 723.676606] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 726.234733] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 731.700185] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 738.376961] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 754.658730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 754.658768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.665191] LustreError: Skipped 1 previous similar message [ 754.666274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 754.666279] Lustre: Skipped 2 previous similar messages [ 754.687794] Lustre: Skipped 2 previous similar messages [ 757.786570] Lustre: server umount lustre-OST0000 complete [ 766.571244] Lustre: server umount lustre-MDT0000 complete [ 769.037337] LustreError: 28406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102609 with bad export cookie 10205460633341353202 [ 769.038839] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.044669] LustreError: 28406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 769.362509] Lustre: server umount lustre-MDT0001 complete [ 777.593817] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 779.638738] Key type lgssc unregistered [ 779.862180] LNet: 32005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.867114] LNetError: 32005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.877088] LNet: Removed LNI 192.168.201.135@tcp [ 780.525483] Key type .llcrypt unregistered [ 780.530972] Key type ._llcrypt unregistered [ 797.157872] Key type ._llcrypt registered [ 797.160208] Key type .llcrypt registered [ 797.255603] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 797.746178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.761166] alg: No test for adler32 (adler32-zlib) [ 798.786538] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 798.989254] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 800.663171] Key type lgssc registered [ 801.361277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 806.894650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 806.906660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.276643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.052027] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 817.507710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.796083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 820.604195] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 824.710194] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 827.408437] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 832.068510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 832.380610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.182685] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 839.960845] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 843.647324] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 853.575871] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:31:32 (1773102692) [ 854.652850] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 855.684414] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 860.254884] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:31:39 (1773102699) [ 866.399654] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:31:45 (1773102705) [ 867.242061] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 873.703319] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 873.706966] Lustre: Skipped 2 previous similar messages [ 875.049715] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:31:53 (1773102713) [ 878.017313] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 878.022169] Lustre: Skipped 3 previous similar messages [ 891.438237] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 891.445946] Lustre: Skipped 6 previous similar messages [ 893.519166] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:32:12 (1773102732) [ 895.688997] Lustre: *** cfs_fail_loc=131b, val=2*** [ 898.091155] Lustre: *** cfs_fail_loc=131b, val=2*** [ 904.178447] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:32:23 (1773102743) [ 907.505925] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 907.508908] Lustre: Skipped 5 previous similar messages [ 915.838351] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:32:34 (1773102754) [ 926.689361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 926.698335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.709905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.785575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.789313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.789566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.801579] Lustre: Skipped 1 previous similar message [ 932.586228] Lustre: server umount lustre-OST0000 complete [ 935.906287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.922379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 940.513016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 940.522889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.541177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.146284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.154862] Lustre: Skipped 3 previous similar messages [ 950.239947] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 950.377136] Lustre: server umount lustre-MDT0000 complete [ 951.272456] LustreError: 34090:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.283904] LustreError: 34090:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 953.469847] LustreError: 33358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102793 with bad export cookie 15332609118762417635 [ 953.471710] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.478548] LustreError: 33358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 953.721652] Lustre: server umount lustre-MDT0001 complete [ 961.545745] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 963.533532] Key type lgssc unregistered [ 963.724675] LNet: 40866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.733179] LNetError: 40866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.756370] LNet: Removed LNI 192.168.201.135@tcp [ 964.270102] Key type .llcrypt unregistered [ 964.271891] Key type ._llcrypt unregistered [ 977.337077] Key type ._llcrypt registered [ 977.339848] Key type .llcrypt registered [ 977.419776] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 987.374872] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 988.077881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.091155] alg: No test for adler32 (adler32-zlib) [ 989.073787] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 989.213799] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 990.859801] Key type lgssc registered [ 991.625510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1016.222678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1016.258116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.430396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1017.464602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1017.536369] Lustre: lustre-MDT0000: new disk, initializing [ 1017.610584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.626422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1020.261474] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1028.728892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.866067] Lustre: 43964:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1028.922683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1028.935934] Lustre: Skipped 1 previous similar message [ 1029.018179] Lustre: lustre-MDT0001: new disk, initializing [ 1029.122369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1029.159471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1029.180348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1032.537331] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1038.211800] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1041.106622] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1045.979983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1046.233802] Lustre: lustre-OST0000: new disk, initializing [ 1046.237094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1046.304553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.983885] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1052.182935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1052.194326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1052.288082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1055.111543] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1057.637990] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1057.762743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1060.213377] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1060.340530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1062.374677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1062.379682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.389611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1062.399137] Lustre: Skipped 1 previous similar message [ 1067.489533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.493074] Lustre: Skipped 2 previous similar messages [ 1067.639368] Lustre: server umount lustre-OST0000 complete [ 1070.052087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1070.057887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.071925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.610211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.610308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.621709] Lustre: Skipped 1 previous similar message [ 1077.730817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1077.739283] Lustre: Skipped 1 previous similar message [ 1084.383857] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1084.534688] Lustre: server umount lustre-MDT0000 complete [ 1087.002767] LustreError: 43957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773102927 with bad export cookie 2165981904520314556 [ 1087.005700] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1087.013054] LustreError: 43957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1087.198843] Lustre: server umount lustre-MDT0001 complete [ 1093.153734] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:35:32 (1773102932) [ 1097.434410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.782386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.320090] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1102.817804] LustreError: 48014:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.827250] LustreError: 48014:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1105.719497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.945747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1108.528572] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1112.441474] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.717657] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1119.242704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1119.545358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.602659] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1127.507744] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1135.014430] LustreError: 50205:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1135.722079] Lustre: Modifying parameter general.timeout=1 in log params [ 1136.948836] Lustre: Modifying parameter general.timeout=3 in log params [ 1136.951939] Lustre: Skipped 1 previous similar message [ 1139.574837] Lustre: Modifying parameter general.timeout=7 in log params [ 1139.578316] Lustre: Skipped 3 previous similar messages [ 1143.677710] Lustre: Modifying parameter general.timeout=14 in log params [ 1143.685000] Lustre: Skipped 6 previous similar messages [ 1152.265936] Lustre: Modifying parameter general.timeout=27 in log params [ 1152.271233] Lustre: Skipped 12 previous similar messages [ 1168.382674] Lustre: Modifying parameter general.timeout=53 in log params [ 1168.385905] Lustre: Skipped 25 previous similar messages [ 1200.846756] Lustre: Modifying parameter general.timeout=110 in log params [ 1200.850136] Lustre: Skipped 56 previous similar messages [ 1217.713205] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:37:36 (1773103056) [ 1219.684314] Lustre: 57130:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1220.876819] Lustre: 57229:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1223.304060] Lustre: 57426:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1223.316919] Lustre: 57426:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1242.193654] Lustre: 58970:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1242.199521] Lustre: 58970:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1242.207856] LustreError: 58970:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1244.512807] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:38:03 (1773103083) [ 1251.812523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.814115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.825017] Lustre: Skipped 2 previous similar messages [ 1251.829279] Lustre: Skipped 3 previous similar messages [ 1253.369800] Lustre: server umount lustre-MDT0000 complete [ 1255.285892] LustreError: 49629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103095 with bad export cookie 2165981904520315438 [ 1255.289163] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.294836] LustreError: 49629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1255.306256] LustreError: 48006:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1256.928687] LustreError: 50166:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.929025] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.513750] Lustre: server umount lustre-MDT0001 complete [ 1259.903475] Lustre: server umount lustre-OST0000 complete [ 1275.672074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.926418] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1275.947699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1275.952530] Lustre: Skipped 42 previous similar messages [ 1276.116573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.500705] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1283.765882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.875100] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1283.890925] Lustre: 61493:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1286.959730] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1290.765857] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1293.384416] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1297.691451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1297.828077] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1301.589946] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1304.915108] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1305.873378] Lustre: 63720:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1310.694552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.700483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1310.703064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.703070] Lustre: Skipped 1 previous similar message [ 1315.583613] Lustre: server umount lustre-OST0000 complete [ 1319.904345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1319.913349] LustreError: Skipped 1 previous similar message [ 1324.221935] Lustre: server umount lustre-MDT0000 complete [ 1326.048566] LustreError: 62708:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.060214] LustreError: 62708:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1326.290403] LustreError: 61485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103166 with bad export cookie 2165981904520318014 [ 1326.291694] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1326.301178] LustreError: 61485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1326.499757] Lustre: server umount lustre-MDT0001 complete [ 1333.214080] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1334.876471] Key type lgssc unregistered [ 1335.067663] LNet: 64911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.074083] LNetError: 64911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.088884] LNet: Removed LNI 192.168.201.135@tcp [ 1335.570821] Key type .llcrypt unregistered [ 1335.572691] Key type ._llcrypt unregistered [ 1350.029142] Key type ._llcrypt registered [ 1350.033231] Key type .llcrypt registered [ 1350.109220] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1350.723630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1350.821147] alg: No test for adler32 (adler32-zlib) [ 1351.740616] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1351.868122] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1353.471233] Key type lgssc registered [ 1354.124567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1358.404838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1358.416459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.719404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.818257] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1366.652943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.049412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.220332] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1371.915951] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1373.813104] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1377.442794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1377.745367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.717286] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1383.632092] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1397.855648] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1402.851983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1402.860754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.866587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1404.714559] Lustre: server umount lustre-MDT0000 complete [ 1406.794381] LustreError: 66263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103247 with bad export cookie 15031763149414008735 [ 1406.799155] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1406.804613] LustreError: 66263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1407.457352] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.458524] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1407.467057] Lustre: Skipped 1 previous similar message [ 1407.468758] LustreError: 66277:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.485477] LustreError: 66277:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1409.064400] Lustre: server umount lustre-MDT0001 complete [ 1411.396217] Lustre: server umount lustre-OST0000 complete [ 1428.213989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.414611] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1428.431265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1428.597170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.082641] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1436.314937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.394512] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1436.418084] Lustre: 70836:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1436.717180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1438.940435] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1442.098808] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1444.050175] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1447.888319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1447.997620] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1448.204293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.317900] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1454.370151] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1455.357137] Lustre: 73072:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1456.489738] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1459.682074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1459.683091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.691811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1459.710030] Lustre: Skipped 1 previous similar message [ 1464.805937] LustreError: 72460:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.818469] LustreError: 72460:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1464.864915] Lustre: server umount lustre-OST0000 complete [ 1467.362231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1467.365443] LustreError: Skipped 1 previous similar message [ 1467.367811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.374609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.377564] Lustre: Skipped 1 previous similar message [ 1473.268303] Lustre: server umount lustre-MDT0000 complete [ 1475.041681] LustreError: 71925:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.052519] LustreError: 71925:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1475.450109] LustreError: 72476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103315 with bad export cookie 15031763149414009554 [ 1475.452478] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1475.455398] LustreError: 72476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1475.650463] Lustre: server umount lustre-MDT0001 complete [ 1481.255661] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1482.797550] Key type lgssc unregistered [ 1482.973247] LNet: 74261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.978129] LNetError: 74261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.986521] LNet: Removed LNI 192.168.201.135@tcp [ 1483.387270] Key type .llcrypt unregistered [ 1483.389300] Key type ._llcrypt unregistered [ 1497.719278] Key type ._llcrypt registered [ 1497.720946] Key type .llcrypt registered [ 1497.778892] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1498.303273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.328375] alg: No test for adler32 (adler32-zlib) [ 1499.246498] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1499.381387] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1500.983175] Key type lgssc registered [ 1501.581391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1505.707533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1505.720941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.048077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.251641] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1513.817459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.182863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1516.257323] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1518.992993] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1520.885586] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1524.451585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1524.746643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.857455] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1530.672524] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1543.600121] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1544.411746] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1544.415250] Lustre: Skipped 1 previous similar message [ 1545.853497] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1545.856721] Lustre: Skipped 2 previous similar messages [ 1547.897940] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1547.901295] Lustre: Skipped 3 previous similar messages [ 1552.143825] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1552.147746] Lustre: Skipped 8 previous similar messages [ 1560.443446] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1560.447067] Lustre: Skipped 16 previous similar messages [ 1576.471723] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1576.476830] Lustre: Skipped 35 previous similar messages [ 1593.477824] Lustre: 83390:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1594.750877] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:43:54 (1773103434) [ 1595.531763] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1596.374184] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:43:55 (1773103435) [ 1598.432288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1598.437228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.444372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1600.481381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1600.482286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1600.484126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.602164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1605.606841] Lustre: Skipped 2 previous similar messages [ 1610.721029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1612.255308] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1612.344035] Lustre: server umount lustre-OST0000 complete [ 1615.840900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.844251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.846906] Lustre: Skipped 1 previous similar message [ 1615.853675] Lustre: Skipped 2 previous similar messages [ 1620.538490] Lustre: server umount lustre-MDT0000 complete [ 1620.962027] LustreError: 76350:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.971075] LustreError: 76350:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1622.392255] LustreError: 75662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103462 with bad export cookie 3274633669575280821 [ 1622.396238] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1622.398282] LustreError: 75662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1622.554545] Lustre: server umount lustre-MDT0001 complete [ 1628.166174] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1629.735125] Key type lgssc unregistered [ 1629.917586] LNet: 84624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.923784] LNetError: 84624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.932588] LNet: Removed LNI 192.168.201.135@tcp [ 1630.430148] Key type .llcrypt unregistered [ 1630.432375] Key type ._llcrypt unregistered [ 1632.418593] Key type ._llcrypt registered [ 1632.420595] Key type .llcrypt registered [ 1632.480543] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1634.814659] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules [ 1635.337977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1635.417071] alg: No test for adler32 (adler32-zlib) [ 1636.323244] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1636.456957] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1638.064128] Key type lgssc registered [ 1638.716754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1639.403876] LustreError: 85745:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 1648.839406] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1653.769107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1660.172155] LustreError: 85745:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1660.188729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.562189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.574139] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1668.108095] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:45:07 (1773103507) [ 1670.472493] LustreError: 86783:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1670.477299] LustreError: 86783:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1670.481877] LustreError: 86783:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 1670.640250] Lustre: server umount lustre-MDT0000 complete [ 1677.078242] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1678.513951] Key type lgssc unregistered [ 1678.679847] LNet: 88238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.685207] LNetError: 88238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.698436] LNet: Removed LNI 192.168.201.135@tcp [ 1679.117868] Key type .llcrypt unregistered [ 1679.119937] Key type ._llcrypt unregistered [ 1689.218857] Key type ._llcrypt registered [ 1689.221489] Key type .llcrypt registered [ 1689.286196] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1689.854268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1689.865387] alg: No test for adler32 (adler32-zlib) [ 1690.775184] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1690.896762] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1692.495919] Key type lgssc registered [ 1693.083329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1697.107148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1697.116366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.416674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.487849] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1704.628263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.096355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1707.260815] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1710.166992] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1712.186853] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1715.747746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1716.131081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.856976] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1721.484256] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1759.240665] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:46:38 (1773103598) [ 1760.048899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1760.904635] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:46:40 (1773103600) [ 1766.370786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.371028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1766.372320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1766.376555] Lustre: Skipped 1 previous similar message [ 1768.428336] Lustre: server umount lustre-MDT0000 complete [ 1770.385652] LustreError: 91042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103610 with bad export cookie 17344458817632962003 [ 1770.388981] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.390612] LustreError: 91042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1770.548559] Lustre: server umount lustre-MDT0001 complete [ 1782.693491] Lustre: server umount lustre-OST0000 complete [ 1788.296749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.528123] LustreError: 93371:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1788.539282] LustreError: 93371:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1788.625390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.697252] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1794.016292] LustreError: 93372:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.943278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.348867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.356175] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1800.050757] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1801.889341] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1808.363741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1808.462613] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1808.466761] LustreError: 93361:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1808.481787] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1808.487109] LustreError: 95228:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1808.490474] LustreError: 95228:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 1808.493830] LustreError: 95228:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 1808.539550] Lustre: server umount lustre-OST0000 complete [ 1808.542346] LustreError: 95228:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1812.812560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.938152] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1812.943207] LustreError: 93360:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1812.947032] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1812.953779] LustreError: 95690:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1812.958099] LustreError: 95690:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 1812.961861] LustreError: 95690:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 1812.997495] Lustre: server umount lustre-OST0000 complete [ 1813.001095] LustreError: 95690:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1817.451067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1817.549350] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1817.552318] Lustre: Found index 0 for lustre-OST0000, updating log [ 1817.555133] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1817.597893] Lustre: lustre-OST0000: new disk, initializing [ 1817.602795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1817.728268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.324838] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1822.051611] Lustre: Failing over lustre-OST0000 [ 1822.153408] Lustre: server umount lustre-OST0000 complete [ 1823.984706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.986704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1823.991444] Lustre: Skipped 1 previous similar message [ 1823.995893] Lustre: Skipped 2 previous similar messages [ 1829.346227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.349275] Lustre: Skipped 2 previous similar messages [ 1830.200322] Lustre: server umount lustre-MDT0000 complete [ 1832.213639] LustreError: 93351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103672 with bad export cookie 17344458817632964747 [ 1832.217219] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1832.219585] LustreError: 93351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1842.733207] Lustre: server umount lustre-MDT0001 complete [ 1846.899965] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:48:06 (1773103686) [ 1854.765393] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1859.775706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.012591] LustreError: 98529:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1860.059100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.001782] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1865.184360] LustreError: 98530:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1866.310228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.638433] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1870.007235] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1873.118243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1873.531440] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1876.202222] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1876.581343] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1876.597488] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1876.597632] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1880.485140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1880.530624] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1880.589487] Lustre: lustre-OST0001: new disk, initializing [ 1880.592787] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1882.471359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1882.479707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1882.539184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1883.772311] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1887.941208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1892.294077] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1900.401270] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.35@tcp (stopping) [ 1900.512257] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1900.516804] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.472886] Lustre: server umount lustre-MDT0001 complete [ 1905.516782] LustreError: 99603:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.384585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.451808] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1910.454701] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1910.477419] Lustre: 98517:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1910.856152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1910.859943] Lustre: Skipped 3 previous similar messages [ 1911.987355] LustreError: 102405:0:(obd_mount.c:212:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1911.993734] LustreError: 102405:0:(tgt_mount.c:667:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1911.999331] LustreError: 98521:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1912.046647] LustreError: 102410:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.201.135@tcp: cfg command failed: rc = -17 [ 1912.052715] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1912.052715] [ 1912.799348] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1915.881187] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1917.335697] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 20:49:16 (1773103756) [ 1925.280304] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1933.460683] Lustre: 104361:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1944.194318] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1945.847769] Lustre: 105493:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2008.034316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2008.036403] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.039402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.046433] Lustre: Skipped 5 previous similar messages [ 2008.049942] Lustre: Skipped 6 previous similar messages [ 2013.187297] Lustre: server umount lustre-MDT0000 complete [ 2014.835814] LustreError: 98510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773103855 with bad export cookie 17344458817632965580 [ 2014.837785] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.842591] LustreError: 98510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2015.005699] Lustre: server umount lustre-MDT0001 complete [ 2027.554586] Lustre: server umount lustre-OST0000 complete [ 2039.666404] Lustre: server umount lustre-OST0001 complete [ 2042.230519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2046.583994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2050.540380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2055.161391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2063.911671] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2068.521509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.535142] Lustre: lustre-MDT0000: reset Object Index mappings [ 2068.763811] LustreError: 109421:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.771907] LustreError: 109421:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2068.813449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.614579] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2074.274208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.285186] Lustre: lustre-MDT0001: reset Object Index mappings [ 2076.347906] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2077.633299] Lustre: 110537:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2080.517168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2083.054849] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2083.884716] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 2086.593915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2088.820257] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 2089.017806] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2093.932922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 2093.933095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 2098.153781] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2099.664075] Lustre: 112388:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2126.531303] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 20:52:45 (1773103965) [ 2155.491752] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.493027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2155.499821] Lustre: Skipped 2 previous similar messages [ 2155.501939] Lustre: Skipped 5 previous similar messages [ 2161.656609] Lustre: server umount lustre-MDT0000 complete [ 2163.159154] LustreError: 109403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104003 with bad export cookie 17344458817633012361 [ 2163.163363] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.163662] LustreError: 109403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2165.728151] LustreError: 109417:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.729001] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.736365] LustreError: 109417:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2165.742452] Lustre: Skipped 2 previous similar messages [ 2169.486277] Lustre: server umount lustre-MDT0001 complete [ 2177.292391] Lustre: server umount lustre-OST0000 complete [ 2193.375283] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2193.464209] Lustre: server umount lustre-OST0001 complete [ 2196.121201] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 2200.534089] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2216.147099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.306550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2216.324899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2216.378910] Lustre: lustre-MDT0000: new disk, initializing [ 2216.418459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.422620] Lustre: Skipped 3 previous similar messages [ 2216.433569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2218.316855] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2221.959888] Lustre: server umount lustre-MDT0000 complete [ 2226.045845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.200562] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2226.206882] Lustre: 116597:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2226.215475] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2227.996061] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2230.128792] Lustre: server umount lustre-MDT0000 complete [ 2233.902924] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 20:54:33 (1773104073) [ 2234.659973] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2235.534439] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 20:54:34 (1773104074) [ 2243.211893] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2248.135883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.135737] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2253.915419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.000903] Lustre: 118609:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2254.044581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2254.047282] Lustre: Skipped 1 previous similar message [ 2254.105381] Lustre: lustre-MDT0001: new disk, initializing [ 2254.182652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2254.189975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2255.962181] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2258.674380] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2261.758941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2261.905976] Lustre: lustre-OST0000: new disk, initializing [ 2261.908531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2263.374828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2263.380325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2263.433705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2264.804769] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2270.436046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2270.522984] Lustre: lustre-OST0001: new disk, initializing [ 2270.526178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2272.076294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2272.082568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2272.136489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2273.128975] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2278.566911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2280.213496] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2459.123520] Lustre: 127371:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2459.130313] Lustre: 127371:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2459.134277] Lustre: 127371:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2459.137846] Lustre: 127371:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2459.141866] Lustre: 127371:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2459.145726] Lustre: 127371:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2582.725178] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:00:21 (1773104421) [ 2594.787274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.790390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.793136] Lustre: Skipped 3 previous similar messages [ 2594.799525] Lustre: Skipped 14 previous similar messages [ 2596.899190] Lustre: server umount lustre-MDT0000 complete [ 2598.545970] LustreError: 119333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773104438 with bad export cookie 17344458817633070447 [ 2598.550763] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.552037] LustreError: 119333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2599.904202] LustreError: 121793:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.912705] LustreError: 121793:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2604.770460] Lustre: server umount lustre-MDT0001 complete [ 2612.489387] Lustre: server umount lustre-OST0000 complete [ 2620.278199] Lustre: server umount lustre-OST0001 complete [ 2623.236615] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 2627.773597] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2644.014335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.128089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2644.144735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2644.192885] Lustre: lustre-MDT0000: new disk, initializing [ 2644.225897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.229576] Lustre: Skipped 5 previous similar messages [ 2644.238308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2645.977385] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2651.142181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.201654] Lustre: 130378:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2651.344446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2653.198304] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2657.071697] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2658.791587] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2661.753369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2663.684863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2663.689891] Lustre: Skipped 1 previous similar message [ 2663.692038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2663.731775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2664.588473] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2668.463296] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2672.672544] Lustre: lustre-MDD0000: changelog on [ 2673.773681] Lustre: lustre-MDD0001: changelog on [ 3228.090195] Lustre: lustre-MDD0001: changelog off [ 3228.946657] Lustre: 130387:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3229.357200] Lustre: lustre-MDD0000: changelog off [ 3232.226126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3232.227728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.229926] LustreError: Skipped 1 previous similar message [ 3232.235883] Lustre: Skipped 2 previous similar messages [ 3232.237316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.245521] Lustre: Skipped 5 previous similar messages [ 3237.598578] Lustre: server umount lustre-OST0000 complete [ 3240.416448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3245.351087] Lustre: server umount lustre-MDT0000 complete [ 3247.019794] LustreError: 131157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105087 with bad export cookie 17344458817633873725 [ 3247.021135] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.025653] LustreError: 131157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3247.195437] Lustre: server umount lustre-MDT0001 complete [ 3251.870898] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3253.204888] Key type lgssc unregistered [ 3253.353555] LNet: 140601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3253.357879] LNetError: 140601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3253.367367] LNet: Removed LNI 192.168.201.135@tcp [ 3253.753140] Key type .llcrypt unregistered [ 3253.754745] Key type ._llcrypt unregistered [ 3261.716186] Key type ._llcrypt registered [ 3261.717273] Key type .llcrypt registered [ 3261.760626] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 3269.385818] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3269.853316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3269.896090] alg: No test for adler32 (adler32-zlib) [ 3270.780644] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 3270.897503] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3272.495147] Key type lgssc registered [ 3272.957984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3287.367404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3287.376227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.498186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3288.516438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3288.567252] Lustre: lustre-MDT0000: new disk, initializing [ 3288.606440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3288.615831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3290.309070] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3294.937620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.989710] Lustre: 143741:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3295.019954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3295.022439] Lustre: Skipped 1 previous similar message [ 3295.066690] Lustre: lustre-MDT0001: new disk, initializing [ 3295.111673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3295.125976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3295.131791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3296.673770] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3300.335423] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3301.803972] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3304.507836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3304.635223] Lustre: lustre-OST0000: new disk, initializing [ 3304.638202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3304.675403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.473321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3305.478264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3305.512958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3306.814552] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3310.219479] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3312.651145] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_module obdecho/obdecho [ 3313.338445] LustreError: 146355:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3313.342743] LustreError: 146355:0:(obd_config.c:880:class_setup()) setup ec failed (-22) [ 3314.026752] LustreError: 146451:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 3320.802386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3320.803268] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.803825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3320.811436] Lustre: Skipped 2 previous similar messages [ 3323.949728] Lustre: server umount lustre-MDT0000 complete [ 3325.473622] LustreError: 143734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105165 with bad export cookie 1066665279619044083 [ 3325.477111] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3325.478359] LustreError: 143734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3325.609069] Lustre: server umount lustre-MDT0001 complete [ 3337.512807] Lustre: server umount lustre-OST0000 complete [ 3345.421354] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 3349.209947] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3363.043930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.163631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3363.177024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3363.223152] Lustre: lustre-MDT0000: new disk, initializing [ 3363.255479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.264025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3364.854381] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3369.770953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.816579] Lustre: 150124:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3369.841379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3369.843118] Lustre: Skipped 1 previous similar message [ 3369.884240] Lustre: lustre-MDT0001: new disk, initializing [ 3369.921794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3369.939164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3369.944104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3371.468332] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3375.111080] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3376.540827] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3379.111831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3379.229809] Lustre: lustre-OST0000: new disk, initializing [ 3379.233782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3379.276472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.449385] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3384.946588] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3386.524991] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3387.899738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3387.904178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3387.941207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3388.659336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3390.227269] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3390.309433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3392.993035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3392.993123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3392.997544] LustreError: Skipped 1 previous similar message [ 3393.003150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3393.006560] Lustre: Skipped 3 previous similar messages [ 3397.145820] Lustre: server umount lustre-OST0000 complete [ 3400.671839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3400.674936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.681597] Lustre: Skipped 1 previous similar message [ 3400.683712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.232834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.233925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.768697] Lustre: server umount lustre-MDT0000 complete [ 3406.347140] LustreError: 150116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105246 with bad export cookie 1066665279619045028 [ 3406.350402] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3406.351710] LustreError: 150116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3406.502395] Lustre: server umount lustre-MDT0001 complete [ 3407.912422] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:14:07 (1773105247) [ 3408.530868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3409.284956] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:14:08 (1773105248) [ 3412.423398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.651207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.261480] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3417.705697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.558910] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3421.912955] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3423.418163] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3426.236772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3428.710662] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3430.780448] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3443.697480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3443.701304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3443.708223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.710235] Lustre: Skipped 1 previous similar message [ 3449.962328] Lustre: server umount lustre-MDT0000 complete [ 3451.328547] LustreError: 155755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105291 with bad export cookie 1066665279619045896 [ 3451.329310] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3451.334876] LustreError: 155755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3451.747462] LustreError: 155751:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.757263] LustreError: 155751:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3451.762394] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.35@tcp (stopping) [ 3451.766871] Lustre: Skipped 4 previous similar messages [ 3453.407956] LustreError: 154134:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.408361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.417642] Lustre: Skipped 2 previous similar messages [ 3456.416252] LustreError: 156294:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.531400] Lustre: server umount lustre-MDT0001 complete [ 3459.936438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3464.713219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.813914] LustreError: 157664:0:(llog.c:1643:llog_backup()) MGC192.168.201.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3464.816892] Lustre: 157664:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.201.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3481.056125] LustreError: 157669:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.066972] LustreError: 157669:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3486.175922] LustreError: 157671:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3486.182577] LustreError: 157671:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 3496.416359] LustreError: 157669:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3496.426765] LustreError: 157669:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 3515.871786] LustreError: 157669:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.876836] LustreError: 157669:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 13 previous similar messages [ 3526.111693] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3526.115423] Lustre: 157664:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.201.135@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3548.002482] LustreError: 157671:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.008795] LustreError: 157671:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 18 previous similar messages [ 3552.735235] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3552.737464] Lustre: 154131:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.201.135@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3567.080290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3567.084352] Lustre: Skipped 2 previous similar messages [ 3568.502835] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3571.172743] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3571.841565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.767455] LustreError: 154131:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3595.743607] LustreError: 141966:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d54ed727 x1859235464976256/t0(0) o250->MGC192.168.201.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3597.403112] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3599.445350] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3600.079331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3600.870914] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3601.497128] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3604.185546] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3611.106415] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.106731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3611.107878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.107882] Lustre: Skipped 3 previous similar messages [ 3611.111722] Lustre: Skipped 1 previous similar message [ 3615.522352] Lustre: server umount lustre-MDT0000 complete [ 3616.225414] LustreError: 157692:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3616.231242] LustreError: 157692:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 38 previous similar messages [ 3616.774961] LustreError: 154119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105457 with bad export cookie 1066665279619052371 [ 3616.780880] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3616.890735] Lustre: server umount lustre-MDT0001 complete [ 3628.443064] Lustre: server umount lustre-OST0000 complete [ 3636.405483] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 3639.876206] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3653.531021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.662042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3653.677215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3653.721667] Lustre: lustre-MDT0000: new disk, initializing [ 3653.756587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.760817] Lustre: Skipped 1 previous similar message [ 3653.768967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3655.232428] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3659.805940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.857970] Lustre: 162483:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3659.968781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3661.487059] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3664.673969] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3665.824435] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3668.017454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3670.241347] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3673.499397] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3674.977272] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3677.182614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3677.217094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3678.103650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3679.520152] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3679.597764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3682.275907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3682.277822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.279507] LustreError: Skipped 1 previous similar message [ 3682.284891] Lustre: Skipped 1 previous similar message [ 3682.285979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3682.290879] Lustre: Skipped 3 previous similar messages [ 3686.426572] Lustre: server umount lustre-OST0000 complete [ 3702.239071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3702.318052] Lustre: server umount lustre-MDT0000 complete [ 3703.794875] LustreError: 164225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105544 with bad export cookie 1066665279619069199 [ 3703.798657] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3703.799750] LustreError: 164225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3703.937853] Lustre: server umount lustre-MDT0001 complete [ 3707.385164] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:19:06 (1773105546) [ 3709.901980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.813219] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3715.289748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.089260] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3719.161772] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3720.652139] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3723.252590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3725.739394] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3727.826434] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3743.132275] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:19:42 (1773105582) [ 3745.249087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3745.253564] LustreError: Skipped 1 previous similar message [ 3750.370768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.374919] Lustre: Skipped 10 previous similar messages [ 3750.465687] Lustre: server umount lustre-OST0000 complete [ 3755.488904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.494996] Lustre: Skipped 6 previous similar messages [ 3758.245364] Lustre: server umount lustre-MDT0000 complete [ 3759.778797] LustreError: 166493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105600 with bad export cookie 1066665279619070081 [ 3759.780248] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3759.784843] LustreError: 166493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3759.949524] Lustre: server umount lustre-MDT0001 complete [ 3764.088323] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3765.347504] Key type lgssc unregistered [ 3765.507307] LNet: 169905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3765.510912] LNetError: 169905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3765.516346] LNet: Removed LNI 192.168.201.135@tcp [ 3765.862541] Key type .llcrypt unregistered [ 3765.864159] Key type ._llcrypt unregistered [ 3774.813073] Key type ._llcrypt registered [ 3774.814414] Key type .llcrypt registered [ 3774.864711] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3775.285125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3775.331854] alg: No test for adler32 (adler32-zlib) [ 3776.201776] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 3776.295416] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3777.887136] Key type lgssc registered [ 3778.321652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3781.454223] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3781.464040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3796.896287] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3802.015179] LustreError: 171108:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.135@tcp: failed processing log, type 4: rc = -110 [ 3807.135325] LustreError: 171108:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3807.140923] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3807.144246] LustreError: MGC192.168.201.135@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3807.148580] LustreError: 171108:0:(tgt_mount.c:1820:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3807.151846] LustreError: 171108:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 3807.154723] LustreError: 171108:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 3807.177674] Lustre: server umount lustre-OST0000 complete [ 3807.179792] LustreError: 171108:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3810.430177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.677468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.222753] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3816.702786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.855879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3818.524926] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3820.772418] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3822.353687] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3825.414501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.513825] LustreError: 173192:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3825.519895] Lustre: 173192:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.201.135@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3825.574876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3828.210341] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3830.522607] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3832.288021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3832.290644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.295854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3832.802855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.804340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3832.808959] Lustre: Skipped 1 previous similar message [ 3837.621406] Lustre: server umount lustre-MDT0000 complete [ 3837.921946] LustreError: 172316:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.931633] LustreError: 172316:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 3839.474251] LustreError: 171586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105679 with bad export cookie 18103444327511593601 [ 3839.478070] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3839.481074] LustreError: 171586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3839.643804] Lustre: server umount lustre-MDT0001 complete [ 3851.691980] Lustre: server umount lustre-OST0000 complete [ 3855.706954] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 3860.130658] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3875.517103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3875.652047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3875.671558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3875.724969] Lustre: lustre-MDT0000: new disk, initializing [ 3875.769031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3875.778372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3877.532852] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3882.465939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.546677] Lustre: 176571:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3882.581673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3882.583985] Lustre: Skipped 1 previous similar message [ 3882.628568] Lustre: lustre-MDT0001: new disk, initializing [ 3882.672874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3882.694075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3882.699694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3884.542230] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3888.288940] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3889.838976] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3893.063905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3893.205229] Lustre: lustre-OST0000: new disk, initializing [ 3893.207854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3893.247301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3893.300841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3893.306833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3893.377755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3895.864067] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3899.858864] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3901.797720] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3901.908078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3904.069653] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3904.173715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3908.576906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3908.584528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.593524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3908.597285] Lustre: Skipped 1 previous similar message [ 3914.210957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3914.212807] Lustre: Skipped 3 previous similar messages [ 3919.327680] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3919.334878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3919.337629] Lustre: Skipped 1 previous similar message [ 3919.416948] Lustre: server umount lustre-OST0000 complete [ 3923.937533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3923.941267] LustreError: Skipped 1 previous similar message [ 3923.943386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.956169] Lustre: Skipped 1 previous similar message [ 3927.417352] Lustre: server umount lustre-MDT0000 complete [ 3929.419760] LustreError: 177351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105769 with bad export cookie 18103444327511594301 [ 3929.424812] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3929.425644] LustreError: 177351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3929.594589] Lustre: server umount lustre-MDT0001 complete [ 3934.159899] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:22:53 (1773105773) [ 3942.936326] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3944.530697] Key type lgssc unregistered [ 3944.702562] LNet: 181091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3944.705609] LNetError: 181091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3944.716640] LNet: Removed LNI 192.168.201.135@tcp [ 3945.192984] Key type .llcrypt unregistered [ 3945.194987] Key type ._llcrypt unregistered [ 3954.757885] Key type ._llcrypt registered [ 3954.760901] Key type .llcrypt registered [ 3954.816850] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3955.267200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3955.301036] alg: No test for adler32 (adler32-zlib) [ 3956.178810] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 3956.282118] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3957.887157] Key type lgssc registered [ 3958.362449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3962.130055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3962.141208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.388548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.133898] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3968.887042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.071462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3970.891644] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3973.198984] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3974.832406] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3978.127385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3978.232178] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3978.234344] LustreError: 183904:0:(llog.c:1684:llog_backup()) MGC192.168.201.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3978.238022] Lustre: 183904:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.201.135@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3978.291470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.743177] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3983.250931] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3984.352555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3984.356476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.357846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3984.363081] Lustre: Skipped 2 previous similar messages [ 3984.365257] Lustre: Skipped 1 previous similar message [ 3989.473349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3990.324195] Lustre: server umount lustre-MDT0000 complete [ 3991.915481] LustreError: 184558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105832 with bad export cookie 7921326906211892772 [ 3991.919110] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3991.923832] LustreError: 184558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3992.068067] Lustre: server umount lustre-MDT0001 complete [ 4003.945786] Lustre: server umount lustre-OST0000 complete [ 4007.720433] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 4012.060942] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4027.842880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4027.987151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4028.008549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4028.063225] Lustre: lustre-MDT0000: new disk, initializing [ 4028.104365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.116209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4030.093485] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4036.790200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.946601] Lustre: 187281:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4036.989824] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4036.994181] Lustre: Skipped 1 previous similar message [ 4036.999111] LustreError: 188047:0:(llog.c:1684:llog_backup()) MGC192.168.201.135@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4037.004509] LustreError: 188047:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 4037.009510] Lustre: 188047:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.201.135@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4037.027854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4037.032682] Lustre: Skipped 1 previous similar message [ 4037.153297] Lustre: lustre-MDT0001: new disk, initializing [ 4037.311206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.392686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4037.404334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4041.146696] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4048.311534] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4052.185723] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4058.978339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4059.253935] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4059.258341] Lustre: Skipped 1 previous similar message [ 4059.261628] LustreError: 188998:0:(llog.c:1684:llog_backup()) MGC192.168.201.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4059.269420] LustreError: 188998:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 4059.276629] Lustre: 188998:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.201.135@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4059.285049] Lustre: 188998:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4059.303891] Lustre: lustre-OST0000: new disk, initializing [ 4059.307288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4059.395661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4065.325713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4065.345898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4065.468931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4065.941907] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4074.346714] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4079.472910] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4079.688973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4085.028431] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4085.213463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4091.365251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4091.365771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.370989] LustreError: Skipped 1 previous similar message [ 4091.385151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4091.388237] Lustre: Skipped 3 previous similar messages [ 4092.812608] Lustre: server umount lustre-OST0000 complete [ 4096.483094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.490724] Lustre: Skipped 1 previous similar message [ 4096.494085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4096.497366] Lustre: Skipped 1 previous similar message [ 4101.462942] Lustre: server umount lustre-MDT0000 complete [ 4101.602645] LustreError: 187294:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.614156] LustreError: 187294:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4104.061036] LustreError: 187274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773105944 with bad export cookie 7921326906211893472 [ 4104.061349] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4104.066902] LustreError: 187274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4104.337655] Lustre: server umount lustre-MDT0001 complete [ 4110.189662] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:25:49 (1773105949) [ 4119.228997] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4125.394536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4125.650371] LustreError: 191983:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.683759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.801655] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4130.789301] LustreError: 191984:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.359047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4135.023548] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4136.884644] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4141.007146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4144.991723] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4150.288623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.396323] Lustre: lustre-OST0001: new disk, initializing [ 4150.402121] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4154.610191] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4160.547938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4160.556581] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4160.639600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4162.733747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4165.841311] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4172.117502] Lustre: DEBUG MARKER: ADD OST3 [ 4175.398943] loop: module loaded [ 4175.459315] loop0: detected capacity change from 0 to 204800000 [ 4175.693422] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4175.704486] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4175.714846] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4179.454790] loop0: detected capacity change from 0 to 204800000 [ 4181.565347] Lustre: DEBUG MARKER: STOP OST3 [ 4182.791889] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.914630] Lustre: lustre-OST0003: new disk, initializing [ 4182.918742] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4182.987487] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4182.992477] Lustre: Skipped 3 previous similar messages [ 4184.696516] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4184.758097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4184.765201] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4184.778158] Lustre: Failing over lustre-OST0003 [ 4184.786424] LustreError: lustre-OST0003-osc-MDT0000: operation seq_query to node 0@lo failed: rc = -19 [ 4184.794301] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.800940] Lustre: Skipped 1 previous similar message [ 4184.803221] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 4184.843961] Lustre: server umount lustre-OST0003 complete [ 4186.508181] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4186.719157] LustreError: 196404:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4190.075760] LustreError: 193534:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4190.192874] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.871790] Lustre: DEBUG MARKER: START OST3 again [ 4195.171069] LustreError: 193432:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.190355] LustreError: 193432:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4195.553060] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4195.753780] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4196.873682] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4197.108493] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 4197.722271] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4197.723136] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4198.971695] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4215.845235] Lustre: Failing over lustre-OST0003 [ 4215.984669] Lustre: server umount lustre-OST0003 complete [ 4216.298674] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.307082] LustreError: 193433:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.309870] Lustre: Skipped 1 previous similar message [ 4216.319788] LustreError: 193433:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4231.653561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4231.656085] Lustre: Skipped 3 previous similar messages [ 4236.408896] Lustre: server umount lustre-MDT0000 complete [ 4236.774336] LustreError: 191980:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.791684] LustreError: 191980:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 10 previous similar messages [ 4238.438060] LustreError: 191964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106078 with bad export cookie 7921326906211894354 [ 4238.441635] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4238.444154] LustreError: 191964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4241.890979] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.901142] Lustre: Skipped 5 previous similar messages [ 4244.721404] Lustre: server umount lustre-MDT0001 complete [ 4247.360713] Lustre: server umount lustre-OST0000 complete [ 4249.828107] Lustre: server umount lustre-OST0001 complete [ 4253.285063] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 4258.282618] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4281.663360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4281.840632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4281.870238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4281.918493] Lustre: lustre-MDT0000: new disk, initializing [ 4282.008313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4282.019140] Lustre: Skipped 1 previous similar message [ 4282.032249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4284.855597] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4292.117104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.211475] Lustre: 201463:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4292.421706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4295.329417] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4300.758230] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4303.179816] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4307.626715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4307.847141] Lustre: lustre-OST0000: new disk, initializing [ 4307.849561] Lustre: Skipped 1 previous similar message [ 4307.853654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4307.857278] Lustre: Skipped 2 previous similar messages [ 4309.387764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4309.395770] Lustre: Skipped 1 previous similar message [ 4309.403372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4309.487092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4312.194382] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4317.783317] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4320.905326] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4321.050813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4323.850056] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4324.010451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4329.962183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4329.962545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4329.972311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4329.982261] LustreError: Skipped 1 previous similar message [ 4329.992776] Lustre: Skipped 5 previous similar messages [ 4331.376366] Lustre: server umount lustre-OST0000 complete [ 4348.383162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4348.521462] Lustre: server umount lustre-MDT0000 complete [ 4350.434763] LustreError: 202702:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.451408] LustreError: 202702:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 4351.498777] LustreError: 201455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106191 with bad export cookie 7921326906211896636 [ 4351.510288] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4351.796905] Lustre: server umount lustre-MDT0001 complete [ 4354.496968] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 21:29:53 (1773106193) [ 4365.109235] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 4371.166418] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4392.617499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.774086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4392.808640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4392.855258] Lustre: lustre-MDT0000: new disk, initializing [ 4392.918685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4394.962602] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4401.329282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.410793] Lustre: 207630:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4401.614399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4403.920933] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4409.008515] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4411.217768] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4415.532145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4415.752366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.759488] Lustre: Skipped 4 previous similar messages [ 4416.966141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4419.309414] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4424.630508] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4427.387558] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4427.607322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4430.180356] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4430.345089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4432.356118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4432.356175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.361444] LustreError: Skipped 1 previous similar message [ 4432.371330] Lustre: Skipped 3 previous similar messages [ 4432.374988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4432.381360] Lustre: Skipped 7 previous similar messages [ 4437.809496] Lustre: server umount lustre-OST0000 complete [ 4442.594548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4454.367190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4454.498641] Lustre: server umount lustre-MDT0000 complete [ 4456.364623] LustreError: 207620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106296 with bad export cookie 7921326906211897518 [ 4456.372439] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4456.375027] LustreError: 207620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4456.538294] Lustre: server umount lustre-MDT0001 complete [ 4461.100969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.410516] LustreError: 211436:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.872493] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4469.290126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.839721] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4475.061456] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4477.148316] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4481.211598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4484.968905] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4488.464695] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4496.864233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4496.873511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4496.878777] Lustre: Skipped 8 previous similar messages [ 4502.808715] Lustre: server umount lustre-OST0000 complete [ 4519.391320] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4519.494160] Lustre: server umount lustre-MDT0000 complete [ 4521.521264] LustreError: 211417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106361 with bad export cookie 7921326906211898393 [ 4521.523312] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4521.536103] LustreError: 211417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4521.712953] Lustre: server umount lustre-MDT0001 complete [ 4624.510629] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 4626.288436] Key type lgssc unregistered [ 4626.471536] LNet: 214746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4626.477426] LNetError: 214746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4626.491294] LNet: Removed LNI 192.168.201.135@tcp [ 4626.982206] Key type .llcrypt unregistered [ 4626.984245] Key type ._llcrypt unregistered [ 4642.692696] Key type ._llcrypt registered [ 4642.694610] Key type .llcrypt registered [ 4642.782763] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 4652.605225] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4653.286374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4653.301205] alg: No test for adler32 (adler32-zlib) [ 4654.280583] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 4654.419521] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 4656.031170] Key type lgssc registered [ 4656.760684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4676.550417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4676.571086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.770689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4677.797590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4677.853236] Lustre: lustre-MDT0000: new disk, initializing [ 4677.927217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.949368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4680.335084] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4686.970893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.036225] Lustre: 218041:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4687.073525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4687.077409] Lustre: Skipped 1 previous similar message [ 4687.125288] Lustre: lustre-MDT0001: new disk, initializing [ 4687.184217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4687.206821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4687.214316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4689.501943] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4694.357875] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4696.552446] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4700.608408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4700.755420] Lustre: lustre-OST0000: new disk, initializing [ 4700.759054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4700.804473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4704.143382] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4708.868186] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4709.934762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4709.945502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4710.026919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4711.444312] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4711.575299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4713.826471] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4713.946189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4714.976806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4714.982313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.988645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4715.489262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4715.494989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4715.495229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4715.505901] Lustre: Skipped 1 previous similar message [ 4717.047357] Lustre: server umount lustre-OST0000 complete [ 4720.609316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.614217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4720.616503] Lustre: Skipped 1 previous similar message [ 4720.622282] Lustre: Skipped 1 previous similar message [ 4725.035326] Lustre: server umount lustre-MDT0000 complete [ 4727.094535] LustreError: 218034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106567 with bad export cookie 1756521078481954474 [ 4727.098214] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4727.105057] LustreError: 218034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4727.294926] Lustre: server umount lustre-MDT0001 complete [ 4737.414487] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4743.459326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.705826] LustreError: 222552:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.717063] LustreError: 222552:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4743.738316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.749212] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4748.769462] LustreError: 222553:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.262717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.494288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4752.685098] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4754.378148] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4757.612909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4760.571899] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4764.803368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.893982] Lustre: lustre-OST0001: new disk, initializing [ 4764.897139] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4764.945790] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4764.949462] Lustre: Skipped 1 previous similar message [ 4767.883516] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4774.424494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4774.428674] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4774.477845] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4775.538498] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4787.255347] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4798.701779] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:37:17 (1773106637) [ 4799.970098] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.972256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4799.975547] Lustre: Skipped 3 previous similar messages [ 4799.979932] Lustre: Skipped 3 previous similar messages [ 4801.874621] Lustre: server umount lustre-MDT0000 complete [ 4803.812361] LustreError: 222534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106644 with bad export cookie 1756521078481955356 [ 4803.816963] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4803.817017] LustreError: 222534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4803.995921] Lustre: server umount lustre-MDT0001 complete [ 4816.209690] Lustre: server umount lustre-OST0000 complete [ 4828.362406] Lustre: server umount lustre-OST0001 complete [ 4831.560615] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 4836.745902] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4853.306205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.453368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4853.475213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4853.529842] Lustre: lustre-MDT0000: new disk, initializing [ 4853.600275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.618688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4855.523895] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4861.152613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.222691] Lustre: 228703:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4861.387476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4863.321454] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4867.243787] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4868.874507] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4872.351920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4872.508470] Lustre: lustre-OST0000: new disk, initializing [ 4872.510931] Lustre: Skipped 1 previous similar message [ 4872.513830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4872.517414] Lustre: Skipped 2 previous similar messages [ 4874.560619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4874.564273] Lustre: Skipped 1 previous similar message [ 4874.566862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4874.631408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4875.324573] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4879.262504] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4881.325047] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4881.428387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4883.315613] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4883.415662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4884.966230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4884.966554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.967550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4884.968624] LustreError: Skipped 1 previous similar message [ 4884.977683] Lustre: Skipped 1 previous similar message [ 4890.414478] Lustre: server umount lustre-OST0000 complete [ 4895.200902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.204287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4895.208261] Lustre: Skipped 1 previous similar message [ 4895.212253] Lustre: Skipped 3 previous similar messages [ 4898.418113] Lustre: server umount lustre-MDT0000 complete [ 4900.124934] LustreError: 228695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106740 with bad export cookie 1756521078481957484 [ 4900.127261] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.133457] LustreError: 228695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4900.282932] Lustre: server umount lustre-MDT0001 complete [ 4904.218281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4904.433956] LustreError: 232509:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.481419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4904.484218] Lustre: Skipped 2 previous similar messages [ 4906.420156] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4909.536779] LustreError: 232510:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4910.497360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.608349] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4915.123433] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4916.729067] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4919.773712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4922.444234] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4924.951641] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5068.256779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5068.258405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5068.260650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5068.266352] Lustre: Skipped 1 previous similar message [ 5073.179387] Lustre: server umount lustre-OST0000 complete [ 5080.821684] Lustre: server umount lustre-MDT0000 complete [ 5082.372443] LustreError: 232490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773106922 with bad export cookie 1756521078481958366 [ 5082.376351] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5082.377990] LustreError: 232490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5082.527872] Lustre: server umount lustre-MDT0001 complete [ 5188.988353] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 5190.244378] Key type lgssc unregistered [ 5190.398622] LNet: 235771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5190.402507] LNetError: 235771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5190.413388] LNet: Removed LNI 192.168.201.135@tcp [ 5190.798133] Key type .llcrypt unregistered [ 5190.801315] Key type ._llcrypt unregistered [ 5202.442212] Key type ._llcrypt registered [ 5202.444271] Key type .llcrypt registered [ 5202.498157] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 5210.409743] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5210.821351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5210.836286] alg: No test for adler32 (adler32-zlib) [ 5211.738825] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5211.850607] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 5213.447160] Key type lgssc registered [ 5213.968072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5233.477858] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5238.352594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5238.366414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.504176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5239.520658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5239.567144] Lustre: lustre-MDT0000: new disk, initializing [ 5239.605161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.616135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5241.644809] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5247.043620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.111954] Lustre: 239764:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5247.145782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5247.149363] Lustre: Skipped 1 previous similar message [ 5247.199382] Lustre: lustre-MDT0001: new disk, initializing [ 5247.244018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5247.264697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5247.269793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5249.027771] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5251.769524] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5254.659534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5254.792156] Lustre: lustre-OST0000: new disk, initializing [ 5254.795597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5254.840522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5257.417421] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5258.625533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5258.630500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5258.680105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5262.611373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5262.694520] Lustre: lustre-OST0001: new disk, initializing [ 5262.697736] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5262.749195] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5265.416507] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5269.004160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5269.009177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5269.050121] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5271.543378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5274.648670] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5284.321881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.325651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5284.330206] Lustre: Skipped 2 previous similar messages [ 5284.333517] Lustre: Skipped 1 previous similar message [ 5289.442661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.446548] Lustre: Skipped 5 previous similar messages [ 5289.716285] Lustre: server umount lustre-MDT0000 complete [ 5291.342083] LustreError: 242965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107131 with bad export cookie 2476453787503001110 [ 5291.346021] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.347761] LustreError: 242965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5291.505431] Lustre: server umount lustre-MDT0001 complete [ 5302.796817] Lustre: server umount lustre-OST0000 complete [ 5314.479157] Lustre: server umount lustre-OST0001 complete [ 5320.113163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5323.981893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5327.380051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5331.272121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5339.000579] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5343.364195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.376160] Lustre: lustre-MDT0000: reset Object Index mappings [ 5343.602953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.262209] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5348.832584] LustreError: 247181:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.841892] LustreError: 247181:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5348.914129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.924279] Lustre: lustre-MDT0001: reset Object Index mappings [ 5350.676595] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5351.889496] Lustre: 248289:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5354.479434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5354.623785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5354.625889] Lustre: Skipped 1 previous similar message [ 5356.989229] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5358.756134] LustreError: 248643:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5360.696631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5363.476264] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5367.980707] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5367.981420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5372.442811] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5373.952904] Lustre: 250131:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5383.651564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5383.652554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.657225] Lustre: Skipped 3 previous similar messages [ 5383.661141] Lustre: Skipped 1 previous similar message [ 5386.600018] Lustre: server umount lustre-MDT0000 complete [ 5388.101214] LustreError: 247908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107228 with bad export cookie 2476453787503007886 [ 5388.105276] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5388.107708] LustreError: 247908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5388.246914] Lustre: server umount lustre-MDT0001 complete [ 5400.069194] Lustre: server umount lustre-OST0000 complete [ 5410.789689] Lustre: server umount lustre-OST0001 complete [ 5417.629545] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:47:37 (1773107257) [ 5423.560342] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 5427.220380] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5441.746709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.877745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5441.895795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5441.940216] Lustre: lustre-MDT0000: new disk, initializing [ 5441.975338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5441.978135] Lustre: Skipped 1 previous similar message [ 5441.984963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5443.685607] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5448.609800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.668613] Lustre: 254348:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5448.788654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5450.347502] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5453.864966] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5455.315796] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5458.258977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5458.395555] Lustre: lustre-OST0000: new disk, initializing [ 5458.397816] Lustre: Skipped 1 previous similar message [ 5458.401057] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5458.403474] Lustre: Skipped 2 previous similar messages [ 5460.223329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5460.225647] Lustre: Skipped 1 previous similar message [ 5460.227669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5460.267765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5460.602710] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5464.024791] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5465.968570] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5466.059922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5467.665112] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5467.752451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5470.690374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5470.690741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5470.694238] LustreError: Skipped 1 previous similar message [ 5470.699117] Lustre: Skipped 1 previous similar message [ 5470.700032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5470.706181] Lustre: Skipped 3 previous similar messages [ 5474.654775] Lustre: server umount lustre-OST0000 complete [ 5479.395221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5479.401605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.407682] Lustre: Skipped 1 previous similar message [ 5479.410297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5482.345621] Lustre: server umount lustre-MDT0000 complete [ 5483.936993] LustreError: 254341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107324 with bad export cookie 2476453787503009979 [ 5483.937742] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5483.950493] LustreError: 254341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5484.117990] Lustre: server umount lustre-MDT0001 complete [ 5492.036362] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5496.402305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.595512] LustreError: 258852:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.617576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5496.620925] Lustre: Skipped 2 previous similar messages [ 5498.249053] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5501.920280] LustreError: 258853:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5502.196548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.070320] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5505.536362] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5508.353311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5510.931240] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5514.300781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5514.389755] Lustre: lustre-OST0001: new disk, initializing [ 5514.392899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5516.256166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5516.262150] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5516.313376] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5517.071870] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5522.071828] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5523.500315] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5530.770316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5530.773677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5533.152498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5533.155619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.161239] Lustre: Skipped 1 previous similar message [ 5533.164846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5533.167545] Lustre: Skipped 2 previous similar messages [ 5539.111815] Lustre: server umount lustre-MDT0000 complete [ 5540.588464] LustreError: 258833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107380 with bad export cookie 2476453787503010861 [ 5540.594861] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.596692] LustreError: 258833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5540.737786] Lustre: server umount lustre-MDT0001 complete [ 5552.585195] Lustre: server umount lustre-OST0000 complete [ 5564.344204] Lustre: server umount lustre-OST0001 complete [ 5571.199852] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5575.345792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.519655] LustreError: 264129:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5575.547774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.550785] Lustre: Skipped 3 previous similar messages [ 5577.181719] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5580.810207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.766280] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5584.066946] Lustre: 265236:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5586.810211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5589.348483] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5591.077711] LustreError: 265589:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.083038] LustreError: 265589:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5592.643823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5595.343546] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5600.047608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5600.055247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5604.195026] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5605.704392] Lustre: 267091:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5612.679378] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5612.685774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5616.207739] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:50:55 (1773107455) [ 5630.946295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.949969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5630.952040] Lustre: Skipped 5 previous similar messages [ 5630.954798] Lustre: Skipped 5 previous similar messages [ 5634.024648] Lustre: server umount lustre-MDT0000 complete [ 5635.372732] LustreError: 265614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107475 with bad export cookie 2476453787503017168 [ 5635.375966] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5635.376751] LustreError: 265614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5636.064266] LustreError: 266846:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5636.072281] LustreError: 266846:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5641.594610] Lustre: server umount lustre-MDT0001 complete [ 5649.096797] Lustre: server umount lustre-OST0000 complete [ 5656.742658] Lustre: server umount lustre-OST0001 complete [ 5659.174310] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 5662.900921] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5680.826476] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5684.798289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5684.907788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5684.920832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5684.960802] Lustre: lustre-MDT0000: new disk, initializing [ 5684.995726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5686.503843] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5691.344438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.400449] Lustre: 271270:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5691.523839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5693.147669] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5698.468813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5700.147606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5700.990338] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5705.666438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5705.791187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5705.794361] Lustre: Skipped 6 previous similar messages [ 5707.502921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5707.982817] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5712.902759] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5724.349443] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 21:52:43 (1773107563) [ 5727.711940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5727.714538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.719517] Lustre: Skipped 3 previous similar messages [ 5727.721952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.725616] Lustre: Skipped 6 previous similar messages [ 5731.313930] Lustre: server umount lustre-MDT0000 complete [ 5732.796227] LustreError: 271263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107573 with bad export cookie 2476453787503025036 [ 5732.800635] LustreError: 271263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5732.800735] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5732.945476] Lustre: server umount lustre-MDT0001 complete [ 5744.579939] Lustre: server umount lustre-OST0000 complete [ 5756.272821] Lustre: server umount lustre-OST0001 complete [ 5763.354163] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 5767.165484] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5781.339172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.449457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5781.454152] Lustre: Skipped 2 previous similar messages [ 5781.466087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5781.468639] Lustre: Skipped 4 previous similar messages [ 5781.510517] Lustre: lustre-MDT0000: new disk, initializing [ 5781.511968] Lustre: Skipped 3 previous similar messages [ 5781.551389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5781.554153] Lustre: Skipped 3 previous similar messages [ 5783.193412] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5788.273623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.338241] Lustre: 278306:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5788.483475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5788.487623] Lustre: Skipped 2 previous similar messages [ 5790.152645] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5793.693229] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5795.163216] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5797.912701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5799.476182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5800.422270] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5803.858800] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5804.709122] Lustre: Mounted lustre-client [ 5805.053853] LustreError: 280805:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8a3856fda000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 5805.088148] Lustre: Unmounted lustre-client [ 5809.632262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5809.632636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.635241] LustreError: Skipped 1 previous similar message [ 5809.641213] Lustre: Skipped 3 previous similar messages [ 5809.644418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5809.649249] Lustre: Skipped 5 previous similar messages [ 5810.387892] Lustre: server umount lustre-OST0000 complete [ 5817.969810] Lustre: server umount lustre-MDT0000 complete [ 5819.540779] LustreError: 279086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107659 with bad export cookie 2476453787503026562 [ 5819.544235] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.545407] LustreError: 279086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5819.691452] Lustre: server umount lustre-MDT0001 complete [ 5823.973901] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 5825.327318] Key type lgssc unregistered [ 5867.231398] LNet: 1 local NIs in recovery (showing 1): 192.168.201.135@tcp [ 5867.487387] LNet: 282082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5867.490458] LNetError: 282082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5936.101552] LNet: Removed LNI 192.168.201.135@tcp [ 5936.529123] Key type .llcrypt unregistered [ 5936.531058] Key type ._llcrypt unregistered [ 5944.233284] Key type ._llcrypt registered [ 5944.235080] Key type .llcrypt registered [ 5944.285360] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 5951.686186] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5952.233686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5952.245711] alg: No test for adler32 (adler32-zlib) [ 5953.129155] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 5953.227354] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 5954.807179] Key type lgssc registered [ 5955.264709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5967.650625] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:56:47 (1773107807) [ 5970.371715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5970.381362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.506733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5971.528151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5971.575900] Lustre: lustre-MDT0000: new disk, initializing [ 5971.615885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.627387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5973.345367] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5978.328823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.386094] Lustre: 285242:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5978.416815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5978.420248] Lustre: Skipped 1 previous similar message [ 5978.470090] Lustre: lustre-MDT0001: new disk, initializing [ 5978.508213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5978.523275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5978.527879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5980.367458] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5984.036881] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5985.581914] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5988.601741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.741334] Lustre: lustre-OST0000: new disk, initializing [ 5988.745434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5988.791680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.324350] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5995.041744] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5996.039297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5996.045896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5996.081795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6006.243898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.244762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6006.246147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.248410] Lustre: Skipped 1 previous similar message [ 6006.251315] LustreError: Skipped 1 previous similar message [ 6010.908028] Lustre: server umount lustre-OST0000 complete [ 6014.432501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6014.436888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.445729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6014.448334] Lustre: Skipped 1 previous similar message [ 6016.480588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6016.480841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.733702] Lustre: server umount lustre-MDT0000 complete [ 6020.382398] LustreError: 285234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107860 with bad export cookie 6998583946652135975 [ 6020.385860] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6020.386812] LustreError: 285234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6020.529994] Lustre: server umount lustre-MDT0001 complete [ 6025.324541] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6026.580323] Key type lgssc unregistered [ 6026.746568] LNet: 288612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6026.750597] LNetError: 288612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6026.758578] LNet: Removed LNI 192.168.201.135@tcp [ 6027.125892] Key type .llcrypt unregistered [ 6027.127892] Key type ._llcrypt unregistered [ 6040.179362] Key type ._llcrypt registered [ 6040.181234] Key type .llcrypt registered [ 6040.230474] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 6040.756781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6040.798710] alg: No test for adler32 (adler32-zlib) [ 6041.688764] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 6041.807949] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 6043.407372] Key type lgssc registered [ 6043.909489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6047.219036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6047.226537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.442570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6050.177604] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6053.949479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6054.118919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6055.900820] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6058.236277] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6060.107056] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6063.160899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6063.382477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6066.116333] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6068.414548] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6076.905370] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6077.697463] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6079.968866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6079.973042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.978515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6083.950210] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.35@tcp (stopping) [ 6084.067233] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.073502] Lustre: Skipped 1 previous similar message [ 6085.693645] Lustre: server umount lustre-MDT0000 complete [ 6087.350842] LustreError: 289968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107927 with bad export cookie 2024805057326836556 [ 6087.355427] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6087.356225] LustreError: 289968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6089.059778] LustreError: 289985:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.069559] LustreError: 289985:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6089.076190] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.35@tcp (stopping) [ 6089.081775] Lustre: Skipped 3 previous similar messages [ 6089.184922] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.192864] LustreError: 291601:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.193161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6092.201729] LustreError: 291601:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6092.204847] Lustre: Skipped 1 previous similar message [ 6093.565165] Lustre: server umount lustre-MDT0001 complete [ 6101.225228] Lustre: server umount lustre-OST0000 complete [ 6104.834813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.623030] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6107.850609] Lustre: 293719:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.135@tcp [ 6108.721129] Lustre: server umount MGS complete [ 6112.347848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.542895] LustreError: 294289:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.576456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.289070] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6117.856606] LustreError: 294290:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.105210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.272290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6120.012014] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6123.649633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6126.216100] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6128.357088] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6129.030396] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6130.515977] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6140.437568] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 6142.432698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6142.434417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.436973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6142.441791] Lustre: Skipped 1 previous similar message [ 6143.858780] Lustre: server umount lustre-OST0000 complete [ 6147.552610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.560684] Lustre: Skipped 1 previous similar message [ 6147.680848] Lustre: server umount lustre-MDT0000 complete [ 6149.371609] LustreError: 294270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773107989 with bad export cookie 2024805057326837382 [ 6149.375395] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6149.376553] LustreError: 294270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6149.549093] Lustre: server umount lustre-MDT0001 complete [ 6154.371555] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6155.590354] Key type lgssc unregistered [ 6155.727612] LNet: 297468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6155.731854] LNetError: 297468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6155.742517] LNet: Removed LNI 192.168.201.135@tcp [ 6156.053714] Key type .llcrypt unregistered [ 6156.054691] Key type ._llcrypt unregistered [ 6162.488335] Key type ._llcrypt registered [ 6162.489336] Key type .llcrypt registered [ 6162.528720] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 6169.673954] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 6170.120934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6170.143453] alg: No test for adler32 (adler32-zlib) [ 6171.029627] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 6171.143194] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 6172.735092] Key type lgssc registered [ 6173.135407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6186.861543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6186.867168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.978203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6187.992681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6188.041281] Lustre: lustre-MDT0000: new disk, initializing [ 6188.066903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.074125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6189.563151] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6193.966090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.014267] Lustre: 300564:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6194.037658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6194.039599] Lustre: Skipped 1 previous similar message [ 6194.076960] Lustre: lustre-MDT0001: new disk, initializing [ 6194.111459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6194.124460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6194.128669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6195.607463] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6198.889668] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6200.218079] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6202.806935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6202.928412] Lustre: lustre-OST0000: new disk, initializing [ 6202.930512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6202.963789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6205.114151] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6205.373028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6205.376789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6205.415610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6208.377333] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6209.893863] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6209.971218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6211.606055] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6211.696471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6215.648975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6215.652159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.653422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6215.656283] Lustre: Skipped 1 previous similar message [ 6218.522794] Lustre: server umount lustre-OST0000 complete [ 6220.768932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.770093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6220.776687] Lustre: Skipped 1 previous similar message [ 6220.780576] Lustre: Skipped 1 previous similar message [ 6225.888127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6225.891938] Lustre: Skipped 1 previous similar message [ 6226.148579] Lustre: server umount lustre-MDT0000 complete [ 6227.631415] LustreError: 302305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773108067 with bad export cookie 10897899350411743258 [ 6227.634214] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6227.635474] LustreError: 302305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6227.784496] Lustre: server umount lustre-MDT0001 complete [ 6230.978206] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:01:10 (1773108070) [ 6237.181872] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 6241.020405] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 6253.638685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.741916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6253.755231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6253.797115] Lustre: lustre-MDT0000: new disk, initializing [ 6253.824516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6253.830701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6255.299368] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6259.730063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6259.777773] Lustre: 306883:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6259.805130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6259.808318] Lustre: Skipped 1 previous similar message [ 6259.847210] Lustre: lustre-MDT0001: new disk, initializing [ 6259.883881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6259.899610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6259.903904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6261.296793] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6264.557899] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6265.744507] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6268.132117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.232264] Lustre: lustre-OST0000: new disk, initializing [ 6268.234400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6268.259602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6270.411146] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6273.700768] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6277.111850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6277.118505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6277.148721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6292.453264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6292.453581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.454284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6292.456536] LustreError: Skipped 2 previous similar messages [ 6292.461137] Lustre: Skipped 1 previous similar message [ 6295.581206] Lustre: server umount lustre-OST0000 complete [ 6297.569239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.569981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.574991] Lustre: Skipped 1 previous similar message [ 6297.577951] Lustre: Skipped 1 previous similar message [ 6303.199597] Lustre: server umount lustre-MDT0000 complete [ 6304.429060] LustreError: 308624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773108144 with bad export cookie 10897899350411744126 [ 6304.432261] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6304.434876] LustreError: 308624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6304.548536] Lustre: server umount lustre-MDT0001 complete [ 6308.483733] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6309.623733] Key type lgssc unregistered [ 6309.764432] LNet: 310647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6309.767937] LNetError: 310647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6309.774308] LNet: Removed LNI 192.168.201.135@tcp [ 6310.108923] Key type .llcrypt unregistered [ 6310.110584] Key type ._llcrypt unregistered [ 6345.709733] Key type ._llcrypt registered [ 6345.710784] Key type .llcrypt registered [ 6345.754735] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6345.874577] Key type .llcrypt unregistered [ 6345.875684] Key type ._llcrypt unregistered [ 6376.000040] Key type ._llcrypt registered [ 6376.001675] Key type .llcrypt registered [ 6376.046415] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6376.174199] Key type .llcrypt unregistered [ 6376.175152] Key type ._llcrypt unregistered [ 6388.633467] Key type ._llcrypt registered [ 6388.634578] Key type .llcrypt registered [ 6388.681046] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6388.793533] Key type .llcrypt unregistered [ 6388.795252] Key type ._llcrypt unregistered [ 6401.116811] Key type ._llcrypt registered [ 6401.117745] Key type .llcrypt registered [ 6401.159889] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6401.285231] Key type .llcrypt unregistered [ 6401.287532] Key type ._llcrypt unregistered [ 6431.787422] Key type ._llcrypt registered [ 6431.789314] Key type .llcrypt registered [ 6431.828880] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6431.946351] Key type .llcrypt unregistered [ 6431.947608] Key type ._llcrypt unregistered [ 6444.277450] Key type ._llcrypt registered [ 6444.278501] Key type .llcrypt registered [ 6444.318589] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6444.421219] Key type .llcrypt unregistered [ 6444.422197] Key type ._llcrypt unregistered [ 6456.026116] Key type ._llcrypt registered [ 6456.027650] Key type .llcrypt registered [ 6456.069528] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6456.180323] Key type .llcrypt unregistered [ 6456.181529] Key type ._llcrypt unregistered [ 6467.506527] Key type ._llcrypt registered [ 6467.507722] Key type .llcrypt registered [ 6467.549068] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6467.663368] Key type .llcrypt unregistered [ 6467.664603] Key type ._llcrypt unregistered [ 6479.688567] Key type ._llcrypt registered [ 6479.689893] Key type .llcrypt registered [ 6479.732522] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6479.847477] Key type .llcrypt unregistered [ 6479.849248] Key type ._llcrypt unregistered [ 6492.169219] Key type ._llcrypt registered [ 6492.170230] Key type .llcrypt registered [ 6492.212342] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6492.314445] Key type .llcrypt unregistered [ 6492.316139] Key type ._llcrypt unregistered [ 6514.832592] Key type ._llcrypt registered [ 6514.834016] Key type .llcrypt registered [ 6514.871969] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6514.978642] Key type .llcrypt unregistered [ 6514.980224] Key type ._llcrypt unregistered [ 6526.197161] Key type ._llcrypt registered [ 6526.198533] Key type .llcrypt registered [ 6526.235245] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6526.345196] Key type .llcrypt unregistered [ 6526.346147] Key type ._llcrypt unregistered [ 6538.550273] Key type ._llcrypt registered [ 6538.551660] Key type .llcrypt registered [ 6538.597044] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6538.710461] Key type .llcrypt unregistered [ 6538.711480] Key type ._llcrypt unregistered [ 6560.238209] Key type ._llcrypt registered [ 6560.239525] Key type .llcrypt registered [ 6560.277958] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 6566.641503] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 6567.074943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6567.085514] alg: No test for adler32 (adler32-zlib) [ 6567.931221] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 6568.009396] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 6569.591131] Key type lgssc registered [ 6569.954158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6580.076896] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6306 sec ========== 22:06:59 (1773108419) [ 6580.678145] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:07:00 (1773108420) === [ 6581.859412] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:07:01 (1773108421) === [ 6595.488284] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 6596.522741] Key type lgssc unregistered [ 6596.655290] LNet: 332234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6596.657331] LNetError: 332234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6596.663349] LNet: Removed LNI 192.168.201.135@tcp [ 6596.960504] Key type .llcrypt unregistered [ 6596.962087] Key type ._llcrypt unregistered