[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491914305 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003295] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008686] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022229] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026824] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] *** VALIDATE tmpfs *** [ 0.031359] *** VALIDATE proc *** [ 0.032242] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042185] debug: unmapping init [mem 0xffffffff8c059000-0xffffffff8c060fff] [ 0.045173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046659] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.055243] rcu: Hierarchical SRCU implementation. [ 0.057481] smp: Bringing up secondary CPUs ... [ 0.058578] x86: Booting SMP configuration: [ 0.059025] .... node #0, CPUs: #1 #2 #3 [ 0.062576] smp: Brought up 1 node, 4 CPUs [ 0.064015] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.223854] node 0 deferred pages initialised in 156ms [ 0.228098] devtmpfs: initialized [ 0.229450] x86/mm: Memory block size: 128MB [ 0.233472] gcov: version magic: 0x41383552 [ 0.237433] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.241118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244424] pinctrl core: initialized pinctrl subsystem [ 0.246300] [ 0.246888] ************************************************************* [ 0.249018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252012] ** ** [ 0.253009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255015] ** ** [ 0.257010] ** This means that this kernel is built to expose internal ** [ 0.258010] ** IOMMU data structures, which may compromise security on ** [ 0.260012] ** your system. ** [ 0.261011] ** ** [ 0.263013] ** If you see this message and you are not debugging the ** [ 0.264008] ** kernel, report this immediately to your vendor! ** [ 0.266019] ** ** [ 0.269011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.271013] ************************************************************* [ 0.274336] NET: Registered protocol family 16 [ 0.275780] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.279099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.282084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.285615] cpuidle: using governor menu [ 0.287000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.289707] PCI: Using configuration type 1 for base access [ 0.292103] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.301133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.303023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.307034] cryptd: max_cpu_qlen set to 1000 [ 0.309218] ACPI: Added _OSI(Module Device) [ 0.311019] ACPI: Added _OSI(Processor Device) [ 0.313013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.315013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.320290] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.328406] ACPI: Interpreter enabled [ 0.330071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.331011] ACPI: Using IOAPIC for interrupt routing [ 0.333103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.337389] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.348495] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.350044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.353019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.357079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.362460] acpiphp: Slot [2] registered [ 0.364200] acpiphp: Slot [5] registered [ 0.365175] acpiphp: Slot [6] registered [ 0.367183] acpiphp: Slot [7] registered [ 0.368184] acpiphp: Slot [8] registered [ 0.370172] acpiphp: Slot [9] registered [ 0.372170] acpiphp: Slot [10] registered [ 0.374175] acpiphp: Slot [3] registered [ 0.375103] acpiphp: Slot [4] registered [ 0.377112] acpiphp: Slot [11] registered [ 0.378104] acpiphp: Slot [12] registered [ 0.380125] acpiphp: Slot [13] registered [ 0.382100] acpiphp: Slot [14] registered [ 0.383116] acpiphp: Slot [15] registered [ 0.385106] acpiphp: Slot [16] registered [ 0.386104] acpiphp: Slot [17] registered [ 0.388104] acpiphp: Slot [18] registered [ 0.389129] acpiphp: Slot [19] registered [ 0.391161] acpiphp: Slot [20] registered [ 0.393106] acpiphp: Slot [21] registered [ 0.394105] acpiphp: Slot [22] registered [ 0.396168] acpiphp: Slot [23] registered [ 0.397136] acpiphp: Slot [24] registered [ 0.399105] acpiphp: Slot [25] registered [ 0.401181] acpiphp: Slot [26] registered [ 0.402120] acpiphp: Slot [27] registered [ 0.404107] acpiphp: Slot [28] registered [ 0.406104] acpiphp: Slot [29] registered [ 0.407125] acpiphp: Slot [30] registered [ 0.409114] acpiphp: Slot [31] registered [ 0.410069] PCI host bridge to bus 0000:00 [ 0.412020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.414022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.417026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.420021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.422023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.425060] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.428328] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.429951] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.433211] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.442019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.447340] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.448023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.451029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.453023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.457025] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.458000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.460054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.462885] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.466833] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.475017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.480017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.483561] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.488019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.501037] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.523035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.538000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.554020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.589020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.600107] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.613027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.629036] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.678022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.697623] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.707025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.715019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.736019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.749249] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.763027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.771021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.845024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.868485] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.875023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.881019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.896026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.906832] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.908437] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.911428] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.913394] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.915302] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.921126] iommu: Default domain type: Passthrough [ 0.923456] SCSI subsystem initialized [ 0.924272] ACPI: bus type USB registered [ 0.926114] usbcore: registered new interface driver usbfs [ 0.927075] usbcore: registered new interface driver hub [ 0.929074] usbcore: registered new device driver usb [ 0.930304] pps_core: LinuxPPS API ver. 1 registered [ 0.933012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.936055] PTP clock support registered [ 0.938054] EDAC MC: Ver: 3.0.0 [ 0.939531] PCI: Using ACPI for IRQ routing [ 0.940761] NetLabel: Initializing [ 0.942009] NetLabel: domain hash size = 128 [ 0.943006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.944075] NetLabel: unlabeled traffic allowed by default [ 0.946219] vgaarb: loaded [ 0.947234] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.948009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.954402] clocksource: Switched to clocksource kvm-clock [ 1.067133] VFS: Disk quotas dquot_6.6.0 [ 1.068818] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.071429] *** VALIDATE ramfs *** [ 1.072545] *** VALIDATE hugetlbfs *** [ 1.073745] pnp: PnP ACPI init [ 1.075812] pnp: PnP ACPI: found 6 devices [ 1.092863] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.096415] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.098671] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.100869] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.103313] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.105213] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.107966] NET: Registered protocol family 2 [ 1.111266] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.115508] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.119836] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.126277] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.129200] TCP: Hash tables configured (established 65536 bind 65536) [ 1.132110] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.135975] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.139684] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.143474] NET: Registered protocol family 1 [ 1.146499] RPC: Registered named UNIX socket transport module. [ 1.149206] RPC: Registered udp transport module. [ 1.151240] RPC: Registered tcp transport module. [ 1.153294] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.156215] NET: Registered protocol family 44 [ 1.158315] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.160794] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.163176] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.166104] PCI: CLS 0 bytes, default 64 [ 1.167971] Unpacking initramfs... [ 2.629207] debug: unmapping init [mem 0xffff8fd27cc54000-0xffff8fd27ffbffff] [ 2.632989] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.635592] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.638644] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.121931] Initialise system trusted keyrings [ 3.123428] Key type blacklist registered [ 3.125720] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.133206] zbud: loaded [ 3.135975] *** VALIDATE nfs *** [ 3.137231] *** VALIDATE nfs4 *** [ 3.138589] pstore: using deflate compression [ 3.141855] Platform Keyring initialized [ 3.241513] NET: Registered protocol family 38 [ 3.243304] Key type asymmetric registered [ 3.244836] Asymmetric key parser 'x509' registered [ 3.246750] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.250036] io scheduler mq-deadline registered [ 3.251835] io scheduler kyber registered [ 3.253727] io scheduler bfq registered [ 3.256179] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.258848] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.261656] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.264611] ACPI: Power Button [PWRF] [ 3.269993] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.276687] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.289096] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.295543] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.312328] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.340534] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.372086] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.377906] Non-volatile memory driver v1.3 [ 3.379759] Linux agpgart interface v0.103 [ 3.410470] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.413302] vda: detected capacity change from 0 to 74760192 [ 3.430529] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.433422] vdb: detected capacity change from 0 to 1073741824 [ 3.447434] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.450440] vdc: detected capacity change from 0 to 2621440000 [ 3.466161] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.469559] vdd: detected capacity change from 0 to 2621440000 [ 3.488604] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.491832] vde: detected capacity change from 0 to 4294967296 [ 3.506129] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.509302] vdf: detected capacity change from 0 to 4294967296 [ 3.516039] libphy: Fixed MDIO Bus: probed [ 3.527603] usbcore: registered new interface driver usbserial_generic [ 3.530167] usbserial: USB Serial support registered for generic [ 3.532231] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.536775] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.538860] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.541197] mousedev: PS/2 mouse device common for all mice [ 3.543948] rtc_cmos 00:05: RTC can wake from S4 [ 3.546731] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.548435] rtc_cmos 00:05: registered as rtc0 [ 3.554858] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.555971] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.558264] intel_pstate: CPU model not supported [ 3.567548] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.569964] hid: raw HID events driver (C) Jiri Kosina [ 3.573706] usbcore: registered new interface driver usbhid [ 3.575781] usbhid: USB HID core driver [ 3.577365] drop_monitor: Initializing network drop monitor service [ 3.579928] Initializing XFRM netlink socket [ 3.582061] NET: Registered protocol family 10 [ 3.585272] Segment Routing with IPv6 [ 3.586609] NET: Registered protocol family 17 [ 3.588612] mpls_gso: MPLS GSO support [ 3.593364] RAS: Correctable Errors collector initialized. [ 3.595214] AVX version of gcm_enc/dec engaged. [ 3.596956] AES CTR mode by8 optimization enabled [ 3.708990] sched_clock: Marking stable (3708957119, 0)->(4640757272, -931800153) [ 3.712910] registered taskstats version 1 [ 3.715182] Loading compiled-in X.509 certificates [ 3.717073] zswap: loaded using pool lzo/zbud [ 3.744634] Key type big_key registered [ 3.759719] Key type encrypted registered [ 3.761828] ima: No TPM chip found, activating TPM-bypass! [ 3.764345] ima: Allocated hash algorithm: sha1 [ 3.766406] ima: No architecture policies found [ 3.768478] evm: Initialising EVM extended attributes: [ 3.770709] evm: security.selinux [ 3.772088] evm: security.ima [ 3.773302] evm: security.capability [ 3.774674] evm: HMAC attrs: 0x1 [ 3.777429] rtc_cmos 00:05: setting system clock to 2026-08-31 04:50:35 UTC (1788151835) [ 3.784279] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 3.787657] debug: unmapping init [mem 0xffffffff8bd82000-0xffffffff8c058fff] [ 3.796167] Write protecting the kernel read-only data: 28672k [ 3.800695] debug: unmapping init [mem 0xffffffff8a403000-0xffffffff8a5fffff] [ 3.803519] debug: unmapping init [mem 0xffffffff8ad14000-0xffffffff8adfffff] [ 3.854061] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.862631] systemd[1]: Detected virtualization kvm. [ 3.864518] systemd[1]: Detected architecture x86-64. [ 3.866106] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.895735] systemd[1]: No hostname configured. [ 3.897636] systemd[1]: Set hostname to . [ 3.899805] random: systemd: uninitialized urandom read (16 bytes read) [ 3.902329] systemd[1]: Initializing machine ID from random generator. [ 4.034729] random: systemd: uninitialized urandom read (16 bytes read) [ 4.036633] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.040469] random: systemd: uninitialized urandom read (16 bytes read) [ 4.043314] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.046531] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.670628] device-mapper: uevent: version 1.0.3 [ 4.672463] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.453234] virtio_net virtio0 ens2: renamed from eth0 [ 5.476466] scsi host0: ata_piix [ 5.484321] random: fast init done [ 5.518958] scsi host1: ata_piix [ 5.520529] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.522670] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.357958] dracut-initqueue[577]: RTNETLINK answers: File exists [ 10.280310] random: crng init done [ 10.281806] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.827747] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.048349] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.308745] SELinux: Disabled at runtime. [ 12.367148] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.375404] systemd[1]: Detected virtualization kvm. [ 12.377568] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.888735] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.892636] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.897910] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.903416] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.906878] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.914208] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.918843] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ 12.992383] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.446612] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.861899] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.891720] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.032922] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.044236] EDAC sbridge: Ver: 1.1.2 [ 15.546588] Key type dns_resolver registered [ 15.894326] NFS: Registering the id_resolver key type [ 15.896700] Key type id_resolver registered [ 15.898529] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg250-server login: [ 48.194733] libcfs: loading out-of-tree module taints kernel. [ 48.247173] Key type ._llcrypt registered [ 48.249255] Key type .llcrypt registered [ 48.339808] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 70.846513] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 72.911696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.922845] alg: No test for adler32 (adler32-zlib) [ 74.360451] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 75.780897] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 77.856285] Key type lgssc registered [ 80.112567] hrtimer: interrupt took 2785672 ns [ 80.241922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.990881] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.168887] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 172.421282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.461106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.979111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.037235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.246179] Lustre: lustre-MDT0000: new disk, initializing [ 174.386146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.426217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 180.694769] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.936972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.095850] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.140849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.145564] Lustre: Skipped 1 previous similar message [ 198.295453] Lustre: lustre-MDT0001: new disk, initializing [ 198.450524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.527150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.549941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.269450] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.326612] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 221.986453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.389340] Lustre: lustre-OST0000: new disk, initializing [ 222.394575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.410778] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.539252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.708921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.718882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.814923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.198643] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.267876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.461468] Lustre: lustre-OST0001: new disk, initializing [ 245.466310] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.470315] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.587152] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.719893] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.644559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 255.657265] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 255.733127] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 265.986547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.401807] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.480980] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.830297] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing yml_node [ 291.455417] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 294.172911] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 296.910710] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 298.636403] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 00:55:28 EDT 2026 [ 320.508450] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 322.478888] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 341.986617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 341.998907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.013825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.010340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.012939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.040394] Lustre: Skipped 3 previous similar messages [ 345.776541] Lustre: server umount lustre-MDT0000 complete [ 349.846230] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152181 with bad export cookie 8703472150788888221 [ 349.847410] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.858066] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 353.249356] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.269134] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.276218] Lustre: Skipped 3 previous similar messages [ 353.277971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 353.328369] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 354.273562] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.294609] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 356.560293] Lustre: server umount lustre-MDT0001 complete [ 361.271406] Lustre: server umount lustre-OST0000 complete [ 365.341307] Lustre: server umount lustre-OST0001 complete [ 373.097521] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 384.208830] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 420.255656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.561866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 420.614328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 420.780506] Lustre: lustre-MDT0000: new disk, initializing [ 420.913957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.933678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 425.716932] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 437.587990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.762890] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 437.841759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 437.847647] Lustre: Skipped 1 previous similar message [ 438.004626] Lustre: lustre-MDT0001: new disk, initializing [ 438.217372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.267101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 438.302176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.046887] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 458.896900] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 464.782480] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 472.260912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 472.605600] Lustre: lustre-OST0000: new disk, initializing [ 472.611820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 472.618865] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 472.699473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.840604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 473.852493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.991073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 480.392572] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 490.000233] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 494.397155] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.589533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.572439] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.757186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 504.809610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 504.822731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.840897] Lustre: Skipped 1 previous similar message [ 504.842861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.869551] Lustre: Skipped 4 previous similar messages [ 509.927653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.931380] Lustre: Skipped 1 previous similar message [ 516.067674] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 516.214960] Lustre: server umount lustre-OST0000 complete [ 520.163693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 520.164942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.179784] LustreError: Skipped 1 previous similar message [ 520.181273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.181279] Lustre: Skipped 2 previous similar messages [ 520.218700] Lustre: Skipped 1 previous similar message [ 526.315460] Lustre: server umount lustre-MDT0000 complete [ 530.405824] LustreError: 15081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.425705] LustreError: 15081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 530.549602] LustreError: 15870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152362 with bad export cookie 8703472150788889754 [ 530.550920] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.563837] LustreError: 15870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 530.963326] Lustre: server umount lustre-MDT0001 complete [ 537.401724] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:59:27 (1788152367) [ 557.519142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.132632] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.220570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.530944] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 563.682796] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.897152] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.933657] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 573.087661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.792259] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 585.138223] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 591.627978] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 593.490585] Lustre: Failing over lustre-MDT0000 [ 593.891569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 593.906280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.940305] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.978509] Lustre: server umount lustre-MDT0000 complete [ 610.784468] Lustre: 3638:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788152426/real 1788152426] req@ffff8fd301bf6680 x1875013174202240/t0(0) o400->MGC192.168.202.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788152442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 610.842920] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.935470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.025799] LustreError: 3636:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8fd1c25a9880 x1875013174205056/t0(0) o250->MGC192.168.202.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 621.447794] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.818707] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 626.660342] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 626.666699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 626.708364] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.849733] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 642.021193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.021635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 642.022418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 642.022424] Lustre: Skipped 3 previous similar messages [ 642.036235] Lustre: Skipped 2 previous similar messages [ 654.304464] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 654.519097] Lustre: server umount lustre-MDT0000 complete [ 658.612553] LustreError: 19903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152490 with bad export cookie 8703472150788891168 [ 658.618659] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.630489] LustreError: 19903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 659.192085] Lustre: server umount lustre-MDT0001 complete [ 668.095460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.869710] LustreError: 23272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.891078] LustreError: 23272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 669.002025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.015906] Lustre: Skipped 2 previous similar messages [ 673.650602] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 681.693629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.006193] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 695.346204] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 703.023356] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 705.624336] Lustre: Failing over lustre-MDT0000 [ 705.931422] Lustre: server umount lustre-MDT0000 complete [ 706.658941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.672304] Lustre: Skipped 1 previous similar message [ 723.424455] Lustre: 3639:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788152538/real 1788152538] req@ffff8fd1c1af9880 x1875013174232960/t0(0) o400->MGC192.168.202.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788152554 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 723.448414] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.322721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.668920] LustreError: 24002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 733.700488] LustreError: 24002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 734.695355] LustreError: 3636:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8fd301984000 x1875013174236160/t0(0) o250->MGC192.168.202.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 735.153481] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 739.349745] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 740.322741] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 740.324411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 740.339851] Lustre: Skipped 1 previous similar message [ 740.361303] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 748.225215] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 755.685246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 755.697526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.706387] Lustre: Skipped 7 previous similar messages [ 758.227297] Lustre: server umount lustre-MDT0000 complete [ 761.797806] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152593 with bad export cookie 8703472150788892106 [ 761.800733] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.808231] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 762.120695] Lustre: server umount lustre-MDT0001 complete [ 771.412280] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:03:21 (1788152601) [ 786.830367] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 796.284943] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 833.991821] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 843.709655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.924737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 843.945478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 844.016843] Lustre: lustre-MDT0000: new disk, initializing [ 844.086657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.093471] Lustre: Skipped 2 previous similar messages [ 844.103635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 849.170378] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 860.331816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.564660] Lustre: 29644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 861.041453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 867.154544] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 877.747694] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 883.474839] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 890.264771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 890.529323] Lustre: 31613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 892.160737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 892.238915] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 892.285551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 892.304195] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 896.385933] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 905.935322] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 911.570520] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 933.351790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.358182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.364855] Lustre: Skipped 2 previous similar messages [ 933.374184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.377345] Lustre: Skipped 3 previous similar messages [ 937.265604] Lustre: server umount lustre-OST0000 complete [ 946.885658] Lustre: server umount lustre-MDT0000 complete [ 948.712358] LustreError: 29658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.736814] LustreError: 29658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 950.182073] LustreError: 29637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152781 with bad export cookie 8703472150788892512 [ 950.196832] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 950.212679] LustreError: 29637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 950.705733] Lustre: server umount lustre-MDT0001 complete [ 961.702974] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 965.777912] Key type lgssc unregistered [ 966.298581] LNet: 33595:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.307839] LNetError: 33595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.326360] LNet: Removed LNI 192.168.202.150@tcp [ 967.610343] Key type .llcrypt unregistered [ 967.612079] Key type ._llcrypt unregistered [ 994.782470] Key type ._llcrypt registered [ 994.784316] Key type .llcrypt registered [ 994.852289] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 996.082556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 996.240716] alg: No test for adler32 (adler32-zlib) [ 997.291361] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 997.658963] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 999.432176] Key type lgssc registered [ 1000.216842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.955294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1007.972149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.350860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.646625] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1019.876972] LustreError: 35009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.887681] LustreError: 35009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1022.600136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.007195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.853268] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1034.135181] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1040.123177] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1047.146520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1047.634861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.660840] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1053.275231] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1054.759168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1059.988925] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1073.230230] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:08:23 (1788152903) [ 1075.301377] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1077.413102] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1083.499632] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:08:34 (1788152914) [ 1092.425890] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:08:42 (1788152922) [ 1093.145757] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1100.593236] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1100.597843] Lustre: Skipped 2 previous similar messages [ 1102.267822] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:08:52 (1788152932) [ 1106.129541] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1106.134533] Lustre: Skipped 3 previous similar messages [ 1122.101810] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1122.106301] Lustre: Skipped 6 previous similar messages [ 1125.527652] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:09:15 (1788152955) [ 1128.601365] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1131.879869] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1139.958342] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:09:30 (1788152970) [ 1141.791101] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1141.798415] Lustre: Skipped 3 previous similar messages [ 1155.689354] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:09:45 (1788152985) [ 1170.402312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.420875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.452531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1172.453605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.463889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1172.467416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1172.487484] Lustre: Skipped 1 previous similar message [ 1177.581658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1177.591378] Lustre: Skipped 1 previous similar message [ 1182.692237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1182.702110] Lustre: Skipped 1 previous similar message [ 1183.201513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1183.323167] Lustre: server umount lustre-OST0000 complete [ 1186.785853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1186.792509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.804919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1192.935208] LustreError: 35010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1192.948064] Lustre: server umount lustre-MDT0000 complete [ 1192.950342] LustreError: 35010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1196.469078] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153028 with bad export cookie 14250242641735307175 [ 1196.475436] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1196.483697] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1196.876071] Lustre: server umount lustre-MDT0001 complete [ 1207.720726] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1210.538521] Key type lgssc unregistered [ 1210.773350] LNet: 42867:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.781817] LNetError: 42867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.804242] LNet: Removed LNI 192.168.202.150@tcp [ 1211.626489] Key type .llcrypt unregistered [ 1211.628366] Key type ._llcrypt unregistered [ 1230.502405] Key type ._llcrypt registered [ 1230.507099] Key type .llcrypt registered [ 1230.587092] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 1247.679239] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1248.958114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1249.001123] alg: No test for adler32 (adler32-zlib) [ 1250.076434] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1250.333739] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1252.000898] Key type lgssc registered [ 1253.260312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1290.444528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1290.498618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.839930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1291.875687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1291.989544] Lustre: lustre-MDT0000: new disk, initializing [ 1292.108422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.140855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1296.572067] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1307.612109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.786239] Lustre: 46023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1307.892955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1307.904443] Lustre: Skipped 1 previous similar message [ 1308.054749] Lustre: lustre-MDT0001: new disk, initializing [ 1308.180039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1308.232865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1308.243128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1312.962443] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1322.011939] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1327.390445] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1334.978804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1335.272488] Lustre: lustre-OST0000: new disk, initializing [ 1335.277494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1335.287309] Lustre: 47989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1335.393160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.975914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1340.983902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1341.164308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1342.630219] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1351.984511] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1357.610746] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1357.864721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1364.173617] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1364.507782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1367.009883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.011237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.012127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.023326] LustreError: Skipped 1 previous similar message [ 1367.044207] Lustre: Skipped 1 previous similar message [ 1372.129789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1372.138880] Lustre: Skipped 1 previous similar message [ 1377.251588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1377.261777] Lustre: Skipped 1 previous similar message [ 1381.344371] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1381.478193] Lustre: server umount lustre-OST0000 complete [ 1387.491296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.508131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1387.517714] Lustre: Skipped 2 previous similar messages [ 1392.511165] Lustre: server umount lustre-MDT0000 complete [ 1396.451314] LustreError: 46014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153228 with bad export cookie 13082404735182826332 [ 1396.453798] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.459330] LustreError: 46014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1396.814304] Lustre: server umount lustre-MDT0001 complete [ 1406.039721] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:13:56 (1788153236) [ 1412.872363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.456411] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.470406] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1413.548712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.730521] LustreError: 50461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.866194] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1423.842882] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.496373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.242794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1433.313567] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1441.693756] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1448.506148] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1457.238583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1457.711557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.439513] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1473.913404] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1478.928157] LustreError: 52988:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1479.931397] Lustre: Modifying parameter general.timeout=1 in log params [ 1481.405260] Lustre: Modifying parameter general.timeout=2 in log params [ 1484.346620] Lustre: Modifying parameter general.timeout=4 in log params [ 1484.356152] Lustre: Skipped 1 previous similar message [ 1489.065257] Lustre: Modifying parameter general.timeout=7 in log params [ 1489.082049] Lustre: Skipped 2 previous similar messages [ 1497.444651] Lustre: Modifying parameter general.timeout=16 in log params [ 1497.450789] Lustre: Skipped 8 previous similar messages [ 1513.974427] Lustre: Modifying parameter general.timeout=33 in log params [ 1513.983435] Lustre: Skipped 16 previous similar messages [ 1546.957047] Lustre: Modifying parameter general.timeout=69 in log params [ 1546.959556] Lustre: Skipped 35 previous similar messages [ 1610.224875] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:17:20 (1788153440) [ 1611.246986] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1611.253660] Lustre: Skipped 61 previous similar messages [ 1613.008378] Lustre: 59932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1615.125951] Lustre: 60030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1617.412933] Lustre: 60145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1621.976811] Lustre: 60342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1622.008698] Lustre: 60342:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1649.626106] Lustre: 61797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1649.632212] Lustre: 61797:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1649.652129] LustreError: 61797:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1653.982880] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:18:04 (1788153484) [ 1668.581457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.588365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.606298] Lustre: Skipped 2 previous similar messages [ 1668.616053] Lustre: Skipped 2 previous similar messages [ 1672.909593] Lustre: server umount lustre-MDT0000 complete [ 1673.697576] LustreError: 52816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1673.721304] LustreError: 52816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1676.355096] LustreError: 53585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153508 with bad export cookie 13082404735182827214 [ 1676.357731] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.364413] LustreError: 53585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1676.694916] Lustre: server umount lustre-MDT0001 complete [ 1690.819809] Lustre: server umount lustre-OST0000 complete [ 1719.132411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.562774] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1719.939436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.567297] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1735.549067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.648681] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1735.669814] Lustre: 64327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1740.948423] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1748.490770] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1753.979903] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1759.885786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.147432] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1760.426776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.432538] Lustre: Skipped 1 previous similar message [ 1765.787601] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1773.145948] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1774.966990] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1774.978655] Lustre: Skipped 22 previous similar messages [ 1775.046947] Lustre: 66893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1785.825511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1785.832167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.843100] Lustre: Skipped 1 previous similar message [ 1785.848964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1788.443996] Lustre: server umount lustre-OST0000 complete [ 1796.585636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.606305] Lustre: Skipped 2 previous similar messages [ 1806.816911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.821764] Lustre: Skipped 7 previous similar messages [ 1807.328403] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1807.532253] Lustre: server umount lustre-MDT0000 complete [ 1811.564838] LustreError: 64320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153643 with bad export cookie 13082404735182830231 [ 1811.570413] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1811.580324] LustreError: 64320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1812.129769] Lustre: server umount lustre-MDT0001 complete [ 1821.871223] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1824.721948] Key type lgssc unregistered [ 1825.123404] LNet: 68107:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.134240] LNetError: 68107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.157438] LNet: Removed LNI 192.168.202.150@tcp [ 1826.271856] Key type .llcrypt unregistered [ 1826.273846] Key type ._llcrypt unregistered [ 1848.406814] Key type ._llcrypt registered [ 1848.410442] Key type .llcrypt registered [ 1848.564423] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1849.869022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1849.894442] alg: No test for adler32 (adler32-zlib) [ 1850.976353] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1851.256229] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1852.912181] Key type lgssc registered [ 1853.893733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1862.355538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1862.375050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.950805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.598960] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1873.383841] LustreError: 69512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1873.410342] LustreError: 69512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1878.090347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.996865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.824917] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1893.225789] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1899.638682] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1908.761605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1909.591197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.324339] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1922.881538] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1944.992748] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1954.789186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.803412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.814883] Lustre: Skipped 2 previous similar messages [ 1954.826177] Lustre: Skipped 2 previous similar messages [ 1959.761443] Lustre: server umount lustre-MDT0000 complete [ 1959.913310] LustreError: 70243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.940768] LustreError: 70243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1962.930829] LustreError: 69491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153794 with bad export cookie 3191610259327254973 [ 1962.937740] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1962.940130] LustreError: 69491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1963.330407] Lustre: server umount lustre-MDT0001 complete [ 1977.164498] Lustre: server umount lustre-OST0000 complete [ 2005.425587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.798423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2005.832714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2006.180486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.400652] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2018.985401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.052785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2019.071720] Lustre: 74410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2019.489312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.335798] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2031.063735] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.829986] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2042.503760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.664280] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2043.050685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.648657] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2057.021519] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2059.262292] Lustre: 76984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2061.504813] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2066.917054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2066.934377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.955197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.430411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2072.564593] Lustre: server umount lustre-OST0000 complete [ 2077.666272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.676913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.686493] Lustre: Skipped 2 previous similar messages [ 2077.705577] Lustre: Skipped 3 previous similar messages [ 2082.788276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.795287] Lustre: Skipped 1 previous similar message [ 2082.925874] Lustre: server umount lustre-MDT0000 complete [ 2086.924803] LustreError: 74403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153918 with bad export cookie 3191610259327255813 [ 2086.933457] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2086.934601] LustreError: 74403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2087.142280] Lustre: server umount lustre-MDT0001 complete [ 2098.475146] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2101.226489] Key type lgssc unregistered [ 2101.517187] LNet: 78200:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.539055] LNetError: 78200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.570326] LNet: Removed LNI 192.168.202.150@tcp [ 2103.524535] Key type .llcrypt unregistered [ 2103.526308] Key type ._llcrypt unregistered [ 2127.882546] Key type ._llcrypt registered [ 2127.884688] Key type .llcrypt registered [ 2128.024023] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2129.372848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.406277] alg: No test for adler32 (adler32-zlib) [ 2130.454503] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2130.735919] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2132.424197] Key type lgssc registered [ 2133.533645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2141.223717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2141.268996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.781504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.344686] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2152.931473] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.960973] LustreError: 79617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2156.147439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.603355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2161.758908] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2169.849781] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2175.607950] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2182.768486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2183.363843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.343413] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2198.317668] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2207.168217] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2208.123776] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2209.936841] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2209.943991] Lustre: Skipped 1 previous similar message [ 2211.946364] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2211.958265] Lustre: Skipped 1 previous similar message [ 2216.665120] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2216.676409] Lustre: Skipped 4 previous similar messages [ 2224.842907] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2224.847982] Lustre: Skipped 8 previous similar messages [ 2241.364425] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2241.371465] Lustre: Skipped 16 previous similar messages [ 2274.001673] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2274.010891] Lustre: Skipped 33 previous similar messages [ 2313.896455] Lustre: 87722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2317.882612] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:29:07 (1788154147) [ 2320.626165] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2323.842378] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:29:13 (1788154153) [ 2330.084229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.085063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2330.089216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2330.090846] Lustre: Skipped 1 previous similar message [ 2330.130823] Lustre: Skipped 1 previous similar message [ 2333.872370] Lustre: server umount lustre-OST0000 complete [ 2340.321901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.349761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.360955] Lustre: Skipped 1 previous similar message [ 2344.988280] Lustre: server umount lustre-MDT0000 complete [ 2345.443622] LustreError: 87932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.485095] LustreError: 87932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2348.748685] LustreError: 79596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154180 with bad export cookie 5704730896315408371 [ 2348.758632] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2349.165819] Lustre: server umount lustre-MDT0001 complete [ 2361.301964] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2364.432935] Key type lgssc unregistered [ 2364.730899] LNet: 88973:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.740860] LNetError: 88973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.757971] LNet: Removed LNI 192.168.202.150@tcp [ 2365.732093] Key type .llcrypt unregistered [ 2365.755877] Key type ._llcrypt unregistered [ 2371.083861] Key type ._llcrypt registered [ 2371.090311] Key type .llcrypt registered [ 2371.262159] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2376.958314] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules [ 2378.630826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2378.664868] alg: No test for adler32 (adler32-zlib) [ 2380.053513] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2380.435772] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2382.179951] Key type lgssc registered [ 2383.752426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2386.251733] LustreError: 90232:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2404.206635] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2407.998571] LustreError: 90232:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2413.995116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2414.010317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.826207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.197817] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2425.326237] LustreError: 91250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.357489] LustreError: 91250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2430.234695] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:31:00 (1788154260) [ 2430.447890] LustreError: 91251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.524187] LustreError: 91282:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2435.536928] LustreError: 91282:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2435.542982] LustreError: 91282:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2438.024987] Lustre: server umount lustre-MDT0000 complete [ 2452.794524] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2455.699337] Key type lgssc unregistered [ 2456.105598] LNet: 92774:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.109557] LNetError: 92774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2456.135320] LNet: Removed LNI 192.168.202.150@tcp [ 2457.271624] Key type .llcrypt unregistered [ 2457.281269] Key type ._llcrypt unregistered [ 2476.509615] Key type ._llcrypt registered [ 2476.518984] Key type .llcrypt registered [ 2476.666826] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2477.950374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.210561] alg: No test for adler32 (adler32-zlib) [ 2479.503537] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2479.820966] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2481.544264] Key type lgssc registered [ 2482.635476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.641202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2490.655167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.214668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.578586] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2501.607880] LustreError: 93994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.638766] LustreError: 93994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2505.064307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2505.781884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2510.504619] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2518.838271] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2524.393510] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2531.841176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2532.433128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.876968] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2547.202627] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2603.237854] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:33:53 (1788154433) [ 2604.779756] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2606.650405] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:33:56 (1788154436) [ 2612.707509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2612.709054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.715199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.738197] Lustre: Skipped 2 previous similar messages [ 2617.828000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.832438] Lustre: Skipped 4 previous similar messages [ 2618.452783] Lustre: server umount lustre-MDT0000 complete [ 2622.541307] LustreError: 93974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154454 with bad export cookie 8367143269442275540 [ 2622.556599] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.559832] LustreError: 93974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2622.983718] Lustre: server umount lustre-MDT0001 complete [ 2637.202773] Lustre: server umount lustre-OST0000 complete [ 2649.154934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.611328] LustreError: 98292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.779450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.610459] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2655.206224] LustreError: 98293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.303467] LustreError: 98292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.447047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.992428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2667.363732] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2674.894771] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2680.851585] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2695.983331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.243572] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2696.263297] LustreError: 98281:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2696.359328] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2696.380606] LustreError: 100375:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -98 [ 2696.401665] LustreError: 100375:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 2696.428079] LustreError: 100375:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2696.585705] Lustre: server umount lustre-OST0000 complete [ 2696.590162] LustreError: 100375:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2705.541488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2705.765616] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2705.771724] LustreError: 98281:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2705.801060] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2705.809497] LustreError: 100841:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -98 [ 2705.815722] LustreError: 100841:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 2705.822227] LustreError: 100841:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2705.912678] Lustre: server umount lustre-OST0000 complete [ 2705.920061] LustreError: 100841:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2715.617945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2715.807911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2715.812579] Lustre: Found index 0 for lustre-OST0000, updating log [ 2715.826302] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2715.915460] Lustre: lustre-OST0000: new disk, initializing [ 2715.919797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2715.930686] Lustre: 101375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2716.278803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.770834] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2726.767471] Lustre: Failing over lustre-OST0000 [ 2726.918075] Lustre: server umount lustre-OST0000 complete [ 2732.899085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.908647] Lustre: Skipped 1 previous similar message [ 2732.909717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.919440] Lustre: Skipped 1 previous similar message [ 2738.151341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.269916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.280856] Lustre: Skipped 2 previous similar messages [ 2746.336129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2746.707280] Lustre: server umount lustre-MDT0000 complete [ 2748.390667] LustreError: 99305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.414764] LustreError: 99305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2751.430078] LustreError: 98272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154583 with bad export cookie 8367143269442278284 [ 2751.433645] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.440406] LustreError: 98272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2751.763574] Lustre: server umount lustre-MDT0001 complete [ 2761.079518] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:36:31 (1788154591) [ 2776.912510] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2787.067699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.360868] LustreError: 103726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.513370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.328199] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2801.977082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.971309] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2811.917617] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2818.871855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2819.851618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.854662] Lustre: Skipped 1 previous similar message [ 2819.868187] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2825.088770] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2828.074246] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2828.090712] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2828.114234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2833.195136] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 2833.207832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2833.233819] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2833.238487] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2833.253016] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2833.408057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.500085] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2833.590975] Lustre: lustre-OST0001: new disk, initializing [ 2833.594760] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2833.600351] Lustre: 106043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2840.109351] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2842.258301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2842.271207] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2842.454798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2848.125537] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2854.072112] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2857.956410] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.962637] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2857.979699] Lustre: Skipped 1 previous similar message [ 2857.997926] Lustre: Skipped 2 previous similar messages [ 2872.288435] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2872.486300] Lustre: server umount lustre-MDT0001 complete [ 2873.325525] LustreError: 103727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.354705] LustreError: 103727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2881.472516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.661437] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2881.667989] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2881.768043] Lustre: 103716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2883.773704] LustreError: 107626:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2883.792667] LustreError: 107626:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2883.809923] LustreError: 103718:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2883.925028] LustreError: 107635:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.150@tcp: cfg command failed: rc = -17 [ 2883.937779] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2883.937779] [ 2887.282778] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2887.670138] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2898.583808] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:38:48 (1788154728) [ 2918.256352] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2936.808647] Lustre: 109660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2959.467905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2963.051279] Lustre: 110795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3050.980228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3050.992656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3051.012441] Lustre: Skipped 1 previous similar message [ 3051.019216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3051.026271] Lustre: Skipped 9 previous similar messages [ 3055.918748] Lustre: server umount lustre-MDT0000 complete [ 3056.620956] LustreError: 103722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3056.647166] LustreError: 103722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3060.835340] LustreError: 107539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154892 with bad export cookie 8367143269442279285 [ 3060.839749] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3060.852395] LustreError: 107539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3061.251818] Lustre: server umount lustre-MDT0001 complete [ 3076.815336] Lustre: server umount lustre-OST0000 complete [ 3077.600359] Lustre: 93307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154893/real 1788154893] req@ffff8fd1c51b9180 x1875015695171584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788154909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3077.635392] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.659070] Lustre: Skipped 3 previous similar messages [ 3080.447317] Lustre: server umount lustre-OST0001 complete [ 3087.804271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3098.347695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3107.281118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3117.434643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3138.684793] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3153.012492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.093090] Lustre: lustre-MDT0000: reset Object Index mappings [ 3153.927574] LustreError: 114744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.069990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.079955] Lustre: Skipped 2 previous similar messages [ 3159.943714] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3169.323581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.352199] Lustre: lustre-MDT0001: reset Object Index mappings [ 3176.108419] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3179.921525] Lustre: 115893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3186.886226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3193.927296] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3194.827178] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3198.938160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3204.463697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.756463] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3210.769265] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3210.833089] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3218.214170] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3222.170518] Lustre: 117772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3271.173539] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:45:01 (1788155101) [ 3297.762230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.786125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3297.791359] Lustre: Skipped 7 previous similar messages [ 3304.078496] Lustre: server umount lustre-MDT0000 complete [ 3308.006945] LustreError: 114745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.030364] LustreError: 114745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3308.133674] LustreError: 116274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155139 with bad export cookie 8367143269442326381 [ 3308.136788] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.156256] LustreError: 116274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3314.754096] Lustre: server umount lustre-MDT0001 complete [ 3324.764058] Lustre: server umount lustre-OST0000 complete [ 3335.094885] Lustre: server umount lustre-OST0001 complete [ 3342.904694] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 3354.133806] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3392.238782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.434613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3392.466086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3392.546220] Lustre: lustre-MDT0000: new disk, initializing [ 3392.619962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.627947] Lustre: Skipped 3 previous similar messages [ 3392.650195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3397.749808] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3405.505556] Lustre: server umount lustre-MDT0000 complete [ 3416.106618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.530458] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3416.560761] Lustre: 122037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3416.585415] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3421.768303] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3426.417696] Lustre: server umount lustre-MDT0000 complete [ 3434.815242] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:47:45 (1788155265) [ 3436.372752] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3438.666046] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:47:48 (1788155268) [ 3458.689371] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3470.483882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.150721] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3484.099683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.250868] Lustre: 124085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3484.298099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3484.301340] Lustre: Skipped 1 previous similar message [ 3484.351580] Lustre: lustre-MDT0001: new disk, initializing [ 3484.456724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3484.469595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3489.599534] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3494.072777] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3501.854499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3502.162855] Lustre: lustre-OST0000: new disk, initializing [ 3502.169921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3502.182231] Lustre: 125664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3503.863772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3503.872795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3503.960795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3509.567230] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3522.742649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3523.023632] Lustre: lustre-OST0001: new disk, initializing [ 3523.029249] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3523.035786] Lustre: 126536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3524.780191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3524.808444] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3524.989399] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3529.833711] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3541.814368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3546.542261] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3906.761398] Lustre: 132951:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3906.768641] Lustre: 132951:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3906.774480] Lustre: 132951:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3906.779357] Lustre: 132951:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3906.789786] Lustre: 132951:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3906.803639] Lustre: 132951:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4182.915290] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:00:12 (1788156012) [ 4200.942747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.948803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4200.957935] Lustre: Skipped 7 previous similar messages [ 4200.976856] Lustre: Skipped 9 previous similar messages [ 4204.801164] Lustre: server umount lustre-MDT0000 complete [ 4206.063157] LustreError: 127353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.094649] LustreError: 127353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4208.136253] LustreError: 124836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156039 with bad export cookie 8367143269442393175 [ 4208.142718] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4208.149065] LustreError: 124836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4211.175118] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.176885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4211.193551] Lustre: Skipped 2 previous similar messages [ 4211.206291] Lustre: Skipped 1 previous similar message [ 4214.761378] Lustre: server umount lustre-MDT0001 complete [ 4224.941865] Lustre: server umount lustre-OST0000 complete [ 4235.273994] Lustre: server umount lustre-OST0001 complete [ 4243.283138] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 4255.704135] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 4294.591510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.889246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4294.916561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4295.016393] Lustre: lustre-MDT0000: new disk, initializing [ 4295.098730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.102721] Lustre: Skipped 5 previous similar messages [ 4295.120750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4300.340473] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4310.288903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4310.381983] Lustre: 136012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4310.424114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4310.431573] Lustre: Skipped 1 previous similar message [ 4310.524943] Lustre: lustre-MDT0001: new disk, initializing [ 4310.654071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4310.669666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4315.027939] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4325.144830] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4331.161733] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4337.242149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4337.504827] Lustre: lustre-OST0000: new disk, initializing [ 4337.509840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4337.521612] Lustre: 137981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4337.653935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.660491] Lustre: Skipped 1 previous similar message [ 4339.081990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4339.095927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4339.338984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4345.346730] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4356.708085] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4367.333740] Lustre: lustre-MDD0000: changelog on [ 4370.834714] Lustre: lustre-MDD0001: changelog on [ 6134.597737] Lustre: lustre-MDD0001: changelog off [ 6136.375628] Lustre: 136018:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6137.342684] Lustre: lustre-MDD0000: changelog off [ 6146.529461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6146.540636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.543893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6146.559475] Lustre: Skipped 1 previous similar message [ 6146.583454] Lustre: Skipped 3 previous similar messages [ 6149.915751] Lustre: server umount lustre-OST0000 complete [ 6153.696954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6153.703244] LustreError: Skipped 1 previous similar message [ 6153.709927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.725310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.771561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.296100] Lustre: server umount lustre-MDT0000 complete [ 6161.898450] LustreError: 136023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.929212] LustreError: 136023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6165.244653] LustreError: 136806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157996 with bad export cookie 8367143269443210061 [ 6165.252206] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6165.261093] LustreError: 136806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6165.812461] Lustre: server umount lustre-MDT0001 complete [ 6180.885966] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 6186.664455] Key type lgssc unregistered [ 6187.025548] LNet: 147088:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6187.075871] LNetError: 147088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6187.140887] LNet: Removed LNI 192.168.202.150@tcp [ 6189.031166] Key type .llcrypt unregistered [ 6189.034882] Key type ._llcrypt unregistered [ 6213.818347] Key type ._llcrypt registered [ 6213.823714] Key type .llcrypt registered [ 6213.983870] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 6229.885451] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6230.899855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6230.980183] alg: No test for adler32 (adler32-zlib) [ 6232.082166] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6232.289846] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 6233.936199] Key type lgssc registered [ 6235.399246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6273.988260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6274.016068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.266720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6275.307201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6275.422298] Lustre: lustre-MDT0000: new disk, initializing [ 6275.532033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.559949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6281.173760] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6292.366059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.498554] Lustre: 150292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6292.556100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6292.560209] Lustre: Skipped 1 previous similar message [ 6292.628654] Lustre: lustre-MDT0001: new disk, initializing [ 6292.709770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6292.752911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6292.769539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6297.496457] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6308.620154] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6314.800472] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6322.907061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6323.261690] Lustre: lustre-OST0000: new disk, initializing [ 6323.265547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6323.280326] Lustre: 152260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6323.396560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.879267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6328.896201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6329.045175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6330.171178] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6340.802976] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6347.547439] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_module obdecho/obdecho [ 6349.439681] LustreError: 153266:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6349.453608] LustreError: 153266:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6351.615361] LustreError: 153362:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6364.642652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6364.651226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.665535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6365.155156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6365.185345] Lustre: Skipped 1 previous similar message [ 6365.197728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6365.200507] Lustre: Skipped 2 previous similar messages [ 6370.278505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6370.293731] Lustre: Skipped 2 previous similar messages [ 6371.084088] Lustre: server umount lustre-MDT0000 complete [ 6375.407319] LustreError: 150283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158207 with bad export cookie 15289455715699435087 [ 6375.418765] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6375.425215] LustreError: 150283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6375.730456] Lustre: server umount lustre-MDT0001 complete [ 6388.559481] Lustre: server umount lustre-OST0000 complete [ 6409.920192] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 6421.789127] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6460.458050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.761475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6460.790254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6460.904308] Lustre: lustre-MDT0000: new disk, initializing [ 6461.007243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.027484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6465.790763] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6476.635497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.740855] Lustre: 157078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6476.789375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6476.794622] Lustre: Skipped 1 previous similar message [ 6476.872785] Lustre: lustre-MDT0001: new disk, initializing [ 6476.961563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6477.014730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6477.029040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6482.171308] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6491.376326] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6496.875937] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6503.504781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6503.724946] Lustre: lustre-OST0000: new disk, initializing [ 6503.728946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6503.733877] Lustre: 159045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6503.798576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.588112] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6509.123434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6509.134406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6509.234194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6518.302562] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6522.834248] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6523.035660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6527.019920] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6527.233970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6530.017830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6530.017961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.026332] LustreError: Skipped 1 previous similar message [ 6530.051367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6530.059462] Lustre: Skipped 1 previous similar message [ 6535.149238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6535.157659] Lustre: Skipped 1 previous similar message [ 6543.328334] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6543.444516] Lustre: server umount lustre-OST0000 complete [ 6548.964404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6548.974169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6548.995516] Lustre: Skipped 1 previous similar message [ 6548.999908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6549.002576] Lustre: Skipped 2 previous similar messages [ 6552.750176] Lustre: server umount lustre-MDT0000 complete [ 6555.620993] LustreError: 157091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6555.645578] LustreError: 157091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6555.961046] LustreError: 157072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158387 with bad export cookie 15289455715699436046 [ 6555.964610] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6555.973557] LustreError: 157072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6556.337424] Lustre: server umount lustre-MDT0001 complete [ 6559.069174] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:39:49 (1788158389) [ 6560.221511] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6561.829421] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:39:52 (1788158392) [ 6568.126033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6568.486711] LustreError: 161461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.552525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.601911] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6574.051243] LustreError: 161462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.169791] LustreError: 161461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.210697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6588.781576] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6598.635367] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6604.602531] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6611.645241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6612.132444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6612.146267] Lustre: Skipped 1 previous similar message [ 6618.051844] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6625.407191] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6636.584689] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 6636.591536] Lustre: Skipped 2 previous similar messages [ 6637.532336] Lustre: server umount lustre-MDT0000 complete [ 6639.081523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6639.095241] LustreError: 163426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6639.113550] Lustre: Skipped 3 previous similar messages [ 6639.136895] LustreError: 163426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6641.735411] LustreError: 164041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158473 with bad export cookie 15289455715699436928 [ 6641.742620] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6641.746512] LustreError: 164041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6648.612746] Lustre: server umount lustre-MDT0001 complete [ 6655.640626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6667.813723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6668.072165] LustreError: 165301:0:(llog.c:1655:llog_backup()) MGC192.168.202.150@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6668.077527] Lustre: 165301:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.150@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6671.843500] LustreError: 165306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.873495] LustreError: 165306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6692.325733] LustreError: 165308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.359680] LustreError: 165308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6708.215913] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.50@tcp (not set up) [ 6708.226000] Lustre: Skipped 3 previous similar messages [ 6727.139361] LustreError: 163292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.171062] LustreError: 163292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 6728.672305] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6728.687847] Lustre: 165301:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.150@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6769.675188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6774.443822] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6774.762634] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6783.358606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.785098] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6793.708859] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6799.864231] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6802.110219] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6808.301602] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6810.097428] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6829.539235] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.543384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6829.548551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6829.553492] Lustre: Skipped 2 previous similar messages [ 6829.572648] Lustre: Skipped 2 previous similar messages [ 6835.666876] Lustre: server umount lustre-MDT0000 complete [ 6839.811287] LustreError: 165328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6839.835501] LustreError: 165328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 48 previous similar messages [ 6839.846559] LustreError: 164041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158671 with bad export cookie 15289455715699443382 [ 6839.861296] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6839.864921] LustreError: 164041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6840.273352] Lustre: server umount lustre-MDT0001 complete [ 6854.489703] Lustre: server umount lustre-OST0000 complete [ 6877.838704] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 6888.419460] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6924.857156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.068408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6925.105916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6925.243231] Lustre: lustre-MDT0000: new disk, initializing [ 6925.311612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6925.318601] Lustre: Skipped 1 previous similar message [ 6925.335555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6930.701483] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6940.572078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.687259] Lustre: 170406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6940.958590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6945.737544] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6955.443209] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6961.971533] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6969.935494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.228987] Lustre: lustre-OST0000: new disk, initializing [ 6970.235444] Lustre: Skipped 1 previous similar message [ 6970.237873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6970.250367] Lustre: Skipped 2 previous similar messages [ 6970.259779] Lustre: 172377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6976.859535] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6977.604390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6977.609438] Lustre: Skipped 1 previous similar message [ 6977.614213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6977.749162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6987.193654] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6991.915968] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6992.191723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6996.625811] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6996.813987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7003.619108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7003.620150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.624954] LustreError: Skipped 1 previous similar message [ 7003.627368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.633929] Lustre: Skipped 2 previous similar messages [ 7003.666095] Lustre: Skipped 4 previous similar messages [ 7004.815464] Lustre: server umount lustre-OST0000 complete [ 7014.681790] Lustre: server umount lustre-MDT0000 complete [ 7017.968854] LustreError: 170399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158849 with bad export cookie 15289455715699460210 [ 7017.976846] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7017.983279] LustreError: 170399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7018.242265] Lustre: server umount lustre-MDT0001 complete [ 7027.069079] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:47:37 (1788158857) [ 7033.751838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.361511] LustreError: 174786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.677796] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7049.340412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.173334] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7065.192935] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7072.729061] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7080.484275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.371276] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7097.956231] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7118.714572] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:49:08 (1788158948) [ 7125.475428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7125.481522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7125.495033] LustreError: Skipped 1 previous similar message [ 7125.527524] Lustre: Skipped 3 previous similar messages [ 7128.454878] Lustre: server umount lustre-OST0000 complete [ 7138.610333] Lustre: server umount lustre-MDT0000 complete [ 7142.481133] LustreError: 174768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158974 with bad export cookie 15289455715699461092 [ 7142.482381] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7142.495279] LustreError: 174768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7142.865336] Lustre: server umount lustre-MDT0001 complete [ 7153.323332] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 7156.164272] Key type lgssc unregistered [ 7156.443944] LNet: 178536:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7156.451261] LNetError: 178536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7157.487274] LNet: Removed LNI 192.168.202.150@tcp [ 7158.423671] Key type .llcrypt unregistered [ 7158.425958] Key type ._llcrypt unregistered [ 7175.626977] Key type ._llcrypt registered [ 7175.628951] Key type .llcrypt registered [ 7175.722624] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7176.697112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7176.754723] alg: No test for adler32 (adler32-zlib) [ 7177.797632] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7177.997110] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 7179.657745] Key type lgssc registered [ 7180.762465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7187.831736] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7187.844586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7203.362735] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7208.480883] LustreError: 179778:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 7213.601398] LustreError: 179778:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7213.612706] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7213.618873] LustreError: MGC192.168.202.150@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7213.626923] LustreError: 179778:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7213.636623] LustreError: 179778:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -22 [ 7213.643356] LustreError: 179778:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 7213.718355] Lustre: server umount lustre-OST0000 complete [ 7213.725664] LustreError: 179778:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7221.713568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7222.998468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7227.318768] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7232.490299] LustreError: 180278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7232.508217] LustreError: 180278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7236.239421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.779287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7242.859372] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7252.685710] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7258.200589] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7265.870862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7266.035107] LustreError: 182095:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7266.053245] Lustre: 182095:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.150@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7266.265469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7272.962194] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7280.753199] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7283.172024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7283.175277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.185468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7286.754807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7286.756248] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.494780] Lustre: server umount lustre-MDT0000 complete [ 7291.882594] LustreError: 181005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7291.909859] LustreError: 181005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7293.372945] LustreError: 180259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159125 with bad export cookie 7358983151454181146 [ 7293.383206] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7293.387974] LustreError: 180259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7293.658481] Lustre: server umount lustre-MDT0001 complete [ 7308.285907] Lustre: server umount lustre-OST0000 complete [ 7317.047834] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 7328.235753] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7366.071833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.310825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7366.338614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7366.442810] Lustre: lustre-MDT0000: new disk, initializing [ 7366.510216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.524963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7370.615437] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7380.700898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.939380] Lustre: 185621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7381.055319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7381.069792] Lustre: Skipped 1 previous similar message [ 7381.200472] Lustre: lustre-MDT0001: new disk, initializing [ 7381.334900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7381.411978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7381.428600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7386.220842] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7396.445192] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7402.685796] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7409.375630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7409.590694] Lustre: lustre-OST0000: new disk, initializing [ 7409.594087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7409.598113] Lustre: 187586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7409.681673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.901548] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7416.915466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7416.930622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7417.090045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7426.946446] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7432.528981] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7432.847580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7438.396154] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7438.631424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7442.917789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.923835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7442.952816] Lustre: Skipped 1 previous similar message [ 7442.961213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7442.974366] Lustre: Skipped 2 previous similar messages [ 7447.355213] Lustre: server umount lustre-OST0000 complete [ 7453.152992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.154287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7453.163171] Lustre: Skipped 2 previous similar messages [ 7453.175257] Lustre: Skipped 2 previous similar messages [ 7458.275477] LustreError: 185627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.304158] LustreError: 185627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7458.334748] Lustre: server umount lustre-MDT0000 complete [ 7462.085659] LustreError: 187585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159293 with bad export cookie 7358983151454181867 [ 7462.089311] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7462.096654] LustreError: 187585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7462.594913] Lustre: server umount lustre-MDT0001 complete [ 7472.764579] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:55:02 (1788159302) [ 7493.363881] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 7497.653990] Key type lgssc unregistered [ 7498.073463] LNet: 190532:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7498.085809] LNetError: 190532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7498.104722] LNet: Removed LNI 192.168.202.150@tcp [ 7499.229652] Key type .llcrypt unregistered [ 7499.234560] Key type ._llcrypt unregistered [ 7520.226682] Key type ._llcrypt registered [ 7520.229859] Key type .llcrypt registered [ 7520.386702] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7521.986663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7522.046994] alg: No test for adler32 (adler32-zlib) [ 7523.119777] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7523.407439] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 7525.259241] Key type lgssc registered [ 7526.529171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7536.812367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7536.823928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.273391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.988100] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7547.364253] LustreError: 191753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.377053] LustreError: 191753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7550.349620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.707859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7554.762235] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7562.390794] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7568.291776] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7576.417490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7576.657710] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7576.664182] LustreError: 193616:0:(llog.c:1696:llog_backup()) MGC192.168.202.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7576.678822] Lustre: 193616:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.150@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7576.874608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7583.244913] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7590.914950] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7593.445722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.453477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.467173] Lustre: Skipped 2 previous similar messages [ 7593.476469] Lustre: Skipped 2 previous similar messages [ 7595.563025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7599.623636] Lustre: server umount lustre-MDT0000 complete [ 7600.621280] LustreError: 191753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7600.640461] LustreError: 191753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7604.891903] LustreError: 191731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159436 with bad export cookie 5479171503939920950 [ 7604.909972] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7605.744400] LustreError: 191749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.750349] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7605.791603] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7605.795154] Lustre: Skipped 2 previous similar messages [ 7605.856378] LustreError: 192481:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000013a27a4b x1875020983857536/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7606.110070] Lustre: server umount lustre-MDT0001 complete [ 7613.237668] Lustre: server umount lustre-OST0000 complete [ 7623.027186] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 7634.142975] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7671.632531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7671.988649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7672.029929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7672.174126] Lustre: lustre-MDT0000: new disk, initializing [ 7672.282372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7672.305338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7677.702586] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7688.523764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.684307] Lustre: 197140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7688.806384] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7688.815398] Lustre: Skipped 1 previous similar message [ 7688.822979] LustreError: 197924:0:(llog.c:1696:llog_backup()) MGC192.168.202.150@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7688.836796] LustreError: 197924:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7688.854440] Lustre: 197924:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.150@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7688.901358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7688.909993] Lustre: Skipped 1 previous similar message [ 7689.034177] Lustre: lustre-MDT0001: new disk, initializing [ 7689.199584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7689.303392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7689.333240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7694.770787] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7706.577095] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7713.617716] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7721.526751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.824447] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7721.828141] Lustre: Skipped 1 previous similar message [ 7721.833849] LustreError: 199084:0:(llog.c:1696:llog_backup()) MGC192.168.202.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7721.845507] LustreError: 199084:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7721.855871] Lustre: 199084:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.150@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7721.867733] Lustre: 199084:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7721.898817] Lustre: lustre-OST0000: new disk, initializing [ 7721.912686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7721.920548] Lustre: 199108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7722.053828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7729.362915] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7729.845691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7729.871888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7730.021124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7740.963892] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7747.146861] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7747.482906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7752.486669] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7752.828822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7755.747617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.753269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7755.779459] Lustre: Skipped 1 previous similar message [ 7755.799401] Lustre: Skipped 1 previous similar message [ 7760.874912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7760.886051] Lustre: Skipped 1 previous similar message [ 7761.509300] Lustre: server umount lustre-OST0000 complete [ 7765.986063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7771.105406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7771.112436] Lustre: Skipped 3 previous similar messages [ 7771.956873] Lustre: server umount lustre-MDT0000 complete [ 7775.861393] LustreError: 197133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159607 with bad export cookie 5479171503939921671 [ 7775.869097] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7775.872826] LustreError: 197133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7776.402391] Lustre: server umount lustre-MDT0001 complete [ 7785.319952] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:00:15 (1788159615) [ 7802.507973] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7814.921053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.419770] LustreError: 202238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.477632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7820.778742] LustreError: 202239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.894205] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7831.016523] LustreError: 202239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.030787] LustreError: 202239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7831.294693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.995079] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7839.084822] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7846.357301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.001107] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7862.521775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7862.679842] Lustre: lustre-OST0001: new disk, initializing [ 7862.683263] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7862.692431] Lustre: 204558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7862.781505] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7862.788121] Lustre: Skipped 2 previous similar messages [ 7870.320300] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7870.621591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7870.632715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7870.784601] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7881.349039] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7888.520342] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7891.279719] Lustre: DEBUG MARKER: ADD OST3 [ 7896.159746] loop: module loaded [ 7896.214275] loop0: detected capacity change from 0 to 204800000 [ 7896.436122] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7896.455560] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7896.482311] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7903.576545] Lustre: DEBUG MARKER: STOP OST3 [ 7903.697132] loop0: detected capacity change from 0 to 204800000 [ 7910.392928] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7910.637327] Lustre: lustre-OST0003: new disk, initializing [ 7910.643848] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7910.662852] Lustre: 206619:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7912.284323] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7912.379946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7912.385632] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7912.440208] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7913.376978] Lustre: Failing over lustre-OST0003 [ 7913.625182] Lustre: server umount lustre-OST0003 complete [ 7914.337137] LustreError: 206768:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7917.540132] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7917.567624] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.587941] LustreError: 203737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.600449] Lustre: Skipped 2 previous similar messages [ 7918.173208] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7929.643759] Lustre: DEBUG MARKER: START OST3 again [ 7938.072539] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7938.322082] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7938.329361] Lustre: Skipped 1 previous similar message [ 7938.348777] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7939.821634] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7940.297873] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7940.303221] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7944.305777] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7964.258263] Lustre: Failing over lustre-OST0003 [ 7964.494272] Lustre: server umount lustre-OST0003 complete [ 7966.183349] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7966.185768] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7966.194918] LustreError: Skipped 1 previous similar message [ 7966.200139] LustreError: 204939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.200150] LustreError: 204939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 7966.241957] Lustre: Skipped 1 previous similar message [ 7984.614045] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.625583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7984.637000] Lustre: Skipped 3 previous similar messages [ 7984.641367] Lustre: Skipped 2 previous similar messages [ 7996.385295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7996.602768] Lustre: server umount lustre-MDT0000 complete [ 8000.411571] LustreError: 202218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159832 with bad export cookie 5479171503939922553 [ 8000.413570] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8000.424589] LustreError: 202218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8007.075517] Lustre: server umount lustre-MDT0001 complete [ 8010.919264] Lustre: server umount lustre-OST0000 complete [ 8014.982838] Lustre: server umount lustre-OST0001 complete [ 8021.142743] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 8028.373735] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8067.957025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8068.179638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8068.202270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8068.258876] Lustre: lustre-MDT0000: new disk, initializing [ 8068.349054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.364954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8074.095931] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8084.951836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.074051] Lustre: 212178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8085.119139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8085.123714] Lustre: Skipped 1 previous similar message [ 8085.188390] Lustre: lustre-MDT0001: new disk, initializing [ 8085.317713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8085.334848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8089.257285] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8097.667985] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8103.034559] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8108.831520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8109.069283] Lustre: 214145:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8110.417524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8110.571928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8114.862910] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8123.949750] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8128.776458] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8129.018338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8134.412732] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8134.692189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8141.283676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.284037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.299700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8141.313270] Lustre: Skipped 3 previous similar messages [ 8141.325579] Lustre: Skipped 12 previous similar messages [ 8142.841120] Lustre: server umount lustre-OST0000 complete [ 8152.545637] Lustre: server umount lustre-MDT0000 complete [ 8155.575238] LustreError: 212170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159987 with bad export cookie 5479171503939924849 [ 8155.578495] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8155.588731] LustreError: 212170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8155.906736] Lustre: server umount lustre-MDT0001 complete [ 8159.647571] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:06:29 (1788159989) [ 8175.411634] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 8184.785744] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8214.356500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8214.618721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8214.644180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8214.648170] Lustre: Skipped 1 previous similar message [ 8214.711665] Lustre: lustre-MDT0000: new disk, initializing [ 8214.714397] Lustre: Skipped 1 previous similar message [ 8214.807669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8214.816682] Lustre: Skipped 1 previous similar message [ 8218.495753] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8228.032529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8228.147554] Lustre: 218751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8228.399694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8232.369311] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8242.115851] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8247.006667] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.641851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.934122] Lustre: 220719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8254.613508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8258.406921] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8266.856784] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8271.381658] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8271.611819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8275.667289] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8275.822749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8280.546495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8280.548797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8280.567339] LustreError: Skipped 1 previous similar message [ 8280.583129] Lustre: Skipped 2 previous similar messages [ 8280.592441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8280.595650] Lustre: Skipped 4 previous similar messages [ 8283.464390] Lustre: server umount lustre-OST0000 complete [ 8289.761831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8293.082399] Lustre: server umount lustre-MDT0000 complete [ 8295.908404] LustreError: 219634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8295.925749] LustreError: 219634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 8296.500254] LustreError: 218744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160128 with bad export cookie 5479171503939925731 [ 8296.515172] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8296.516096] LustreError: 218744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8296.772142] Lustre: server umount lustre-MDT0001 complete [ 8303.860087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.622645] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8316.095622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8320.569877] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8327.222602] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8332.667667] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8339.583357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8339.858588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.870660] Lustre: Skipped 7 previous similar messages [ 8346.985776] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8354.569282] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8370.688301] Lustre: server umount lustre-OST0000 complete [ 8381.036710] Lustre: server umount lustre-MDT0000 complete [ 8384.945919] LustreError: 222919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160216 with bad export cookie 5479171503939926613 [ 8384.958800] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8384.966929] LustreError: 222919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8385.356636] Lustre: server umount lustre-MDT0001 complete [ 8492.142594] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 8494.651167] Key type lgssc unregistered [ 8494.927415] LNet: 226604:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8494.935214] LNetError: 226604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8494.954720] LNet: Removed LNI 192.168.202.150@tcp [ 8495.622278] Key type .llcrypt unregistered [ 8495.623796] Key type ._llcrypt unregistered [ 8518.812614] Key type ._llcrypt registered [ 8518.815554] Key type .llcrypt registered [ 8518.908765] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 8532.690654] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8533.841215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8534.017701] alg: No test for adler32 (adler32-zlib) [ 8535.091764] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8535.279622] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 8536.920186] Key type lgssc registered [ 8537.801721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8571.941753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8571.969989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8573.300917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8573.332923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8573.439590] Lustre: lustre-MDT0000: new disk, initializing [ 8573.488396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8573.532515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8577.928157] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8587.700253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.802972] Lustre: 229955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8587.845719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8587.848545] Lustre: Skipped 1 previous similar message [ 8587.923785] Lustre: lustre-MDT0001: new disk, initializing [ 8587.995706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8588.021256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8588.027611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8591.508984] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8600.846370] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8605.714785] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8611.903597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8612.146248] Lustre: lustre-OST0000: new disk, initializing [ 8612.151153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8612.161625] Lustre: 231924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8612.258167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.869648] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8619.070336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8619.079737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8619.196759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8626.319407] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8631.552394] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8631.751209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8636.820461] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8636.997123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8639.970591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8639.982848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8640.015497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8640.023732] Lustre: Skipped 1 previous similar message [ 8645.093569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8645.102674] Lustre: Skipped 1 previous similar message [ 8650.215892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8650.223562] Lustre: Skipped 1 previous similar message [ 8653.280213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8653.478501] Lustre: server umount lustre-OST0000 complete [ 8659.938073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8659.944204] LustreError: Skipped 1 previous similar message [ 8659.949509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.961474] Lustre: Skipped 1 previous similar message [ 8659.966088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8662.917554] Lustre: server umount lustre-MDT0000 complete [ 8666.177368] LustreError: 229946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160497 with bad export cookie 9067096411825812297 [ 8666.178914] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8666.188188] LustreError: 229946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8666.470273] Lustre: server umount lustre-MDT0001 complete [ 8680.788423] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8690.506419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8690.787489] LustreError: 234857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8690.793792] LustreError: 234857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8690.822232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8694.747146] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8696.299747] LustreError: 234858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.388829] LustreError: 234857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8703.961599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.343275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8708.274514] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8711.169399] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8717.814769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8718.092261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8723.893975] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8731.727570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.884461] Lustre: lustre-OST0001: new disk, initializing [ 8731.888482] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8731.894775] Lustre: 237163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8738.500520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8738.509112] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8738.673949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8739.438280] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8750.934832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8763.744967] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8776.955239] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:16:46 (1788160606) [ 8779.749176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.768338] Lustre: Skipped 1 previous similar message [ 8779.770620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8779.788659] Lustre: Skipped 2 previous similar messages [ 8785.488708] Lustre: server umount lustre-MDT0000 complete [ 8789.909698] LustreError: 234838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160621 with bad export cookie 9067096411825813179 [ 8789.925382] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.930813] LustreError: 234838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8789.985941] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.991081] LustreError: 234852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8789.991517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8789.991521] Lustre: Skipped 5 previous similar messages [ 8789.993861] Lustre: Skipped 3 previous similar messages [ 8790.017780] LustreError: 234852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8790.126846] Lustre: server umount lustre-MDT0001 complete [ 8800.644604] Lustre: server umount lustre-OST0000 complete [ 8805.147677] Lustre: server umount lustre-OST0001 complete [ 8811.868941] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 8821.832641] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8865.134650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.385622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8865.408279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8865.476613] Lustre: lustre-MDT0000: new disk, initializing [ 8865.540788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8865.543549] Lustre: Skipped 1 previous similar message [ 8865.556093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8870.799262] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8881.581136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8881.692498] Lustre: 241109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8881.743187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8881.747270] Lustre: Skipped 1 previous similar message [ 8881.809764] Lustre: lustre-MDT0001: new disk, initializing [ 8881.939684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8881.959618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8886.240929] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8895.186190] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8900.952589] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8907.950310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8908.195543] Lustre: lustre-OST0000: new disk, initializing [ 8908.199499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8908.205397] Lustre: 243077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8908.343748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8908.351015] Lustre: Skipped 1 previous similar message [ 8909.595049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8909.611555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8909.719946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8914.144231] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8925.666895] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8930.996580] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8931.225939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8936.013421] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8936.181234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8940.515991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8940.521513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8940.539259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8940.541993] Lustre: Skipped 2 previous similar messages [ 8952.288119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8952.383625] Lustre: server umount lustre-OST0000 complete [ 8955.873675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8955.888841] Lustre: Skipped 2 previous similar messages [ 8970.210652] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8970.439318] Lustre: server umount lustre-MDT0000 complete [ 8971.234394] LustreError: 241114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8971.248041] LustreError: 241114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8973.932566] LustreError: 244619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160805 with bad export cookie 9067096411825815314 [ 8973.944423] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8973.950499] LustreError: 244619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8974.360352] Lustre: server umount lustre-MDT0001 complete [ 8983.437894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8983.982329] LustreError: 245296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8984.051878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.878428] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 8998.372684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9004.467336] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9013.891180] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9021.122104] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9029.115940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9036.851755] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9046.127626] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9197.537524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9197.542027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9197.555106] LustreError: Skipped 1 previous similar message [ 9197.559092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9197.574329] Lustre: Skipped 1 previous similar message [ 9197.600402] Lustre: Skipped 11 previous similar messages [ 9199.303844] Lustre: server umount lustre-OST0000 complete [ 9203.686640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9209.462709] Lustre: server umount lustre-MDT0000 complete [ 9212.910974] LustreError: 245297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.947303] LustreError: 245297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9213.040653] LustreError: 247142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161044 with bad export cookie 9067096411825816196 [ 9213.042926] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9213.058931] LustreError: 247142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9213.674763] Lustre: server umount lustre-MDT0001 complete [ 9316.418879] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 9319.153768] Key type lgssc unregistered [ 9319.424455] LNet: 248907:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9319.442881] LNetError: 248907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9319.467874] LNet: Removed LNI 192.168.202.150@tcp [ 9320.399262] Key type .llcrypt unregistered [ 9320.401171] Key type ._llcrypt unregistered [ 9345.014744] Key type ._llcrypt registered [ 9345.020841] Key type .llcrypt registered [ 9345.169633] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 9360.987981] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9361.873692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9361.898140] alg: No test for adler32 (adler32-zlib) [ 9362.916656] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9363.090461] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 9364.728170] Key type lgssc registered [ 9365.766699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9408.956502] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9418.995860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9419.035278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9420.382568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9420.415526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9420.498673] Lustre: lustre-MDT0000: new disk, initializing [ 9420.593914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9420.634295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9424.917953] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9434.206083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.314175] Lustre: 252977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9434.360961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9434.363983] Lustre: Skipped 1 previous similar message [ 9434.454391] Lustre: lustre-MDT0001: new disk, initializing [ 9434.529746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9434.572158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9434.582830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9438.678293] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9443.435322] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9449.265982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9449.581868] Lustre: lustre-OST0000: new disk, initializing [ 9449.595548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9449.607516] Lustre: 254607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9449.731439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9454.648720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9454.664602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9454.784518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9455.936188] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9466.453808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9466.617446] Lustre: lustre-OST0001: new disk, initializing [ 9466.625807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9466.631789] Lustre: 255476:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9466.738380] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9472.517656] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9472.550209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9472.557990] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9472.635717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9483.613558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9494.276136] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9506.273674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9506.304550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.333644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9508.202609] Lustre: server umount lustre-MDT0000 complete [ 9508.336652] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9508.366748] Lustre: Skipped 2 previous similar messages [ 9512.781317] LustreError: 252968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161344 with bad export cookie 17836123595595650450 [ 9512.793519] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9512.808220] LustreError: 252968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9513.548975] Lustre: server umount lustre-MDT0001 complete [ 9529.181488] Lustre: server umount lustre-OST0000 complete [ 9529.632132] Lustre: 250483:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788161345/real 1788161345] req@ffff8fd2ed86b480 x1875022912237696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788161361 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9529.670878] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.984578] Lustre: server umount lustre-OST0001 complete [ 9550.410532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9561.471545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9570.901418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9580.500262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9601.235741] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9611.040429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9611.060734] Lustre: lustre-MDT0000: reset Object Index mappings [ 9611.450106] LustreError: 260488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9611.467082] LustreError: 260488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9611.535761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9616.116820] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9616.875079] LustreError: 260489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9621.997198] LustreError: 260488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.755672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9625.780961] Lustre: lustre-MDT0001: reset Object Index mappings [ 9626.125033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9630.398473] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9632.975747] Lustre: 261630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9639.591554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9647.436915] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9651.304791] LustreError: 261984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.437379] LustreError: 262515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.456458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9656.460755] LustreError: 262515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9656.684688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9657.005273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9657.014101] Lustre: Skipped 1 previous similar message [ 9662.472013] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9663.255828] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9671.009783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9674.949219] Lustre: 263501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9680.813698] Lustre: server umount lustre-MDT0000 complete [ 9682.412725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9682.426886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.451987] LustreError: 261221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9686.169773] LustreError: 263503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161517 with bad export cookie 17836123595595657240 [ 9686.180304] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9686.192607] LustreError: 263503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9687.262712] Lustre: server umount lustre-MDT0001 complete [ 9701.431738] Lustre: server umount lustre-OST0000 complete [ 9704.289129] Lustre: 250482:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788161519/real 1788161519] req@ffff8fd2ffb1aa00 x1875022912284544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788161535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9704.313479] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.326765] Lustre: Skipped 3 previous similar messages [ 9705.938090] Lustre: server umount lustre-OST0001 complete [ 9727.463726] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:32:37 (1788161557) [ 9747.246696] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 9759.218614] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9792.428630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9792.698725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9792.731933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9792.813975] Lustre: lustre-MDT0000: new disk, initializing [ 9792.870074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9792.883593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9796.493211] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9807.292127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9807.361940] Lustre: 267756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9807.410570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9807.413926] Lustre: Skipped 1 previous similar message [ 9807.493745] Lustre: lustre-MDT0001: new disk, initializing [ 9807.608026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9807.635411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9812.050852] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9822.245378] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9827.921239] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9835.449491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9835.726885] Lustre: lustre-OST0000: new disk, initializing [ 9835.732815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9835.742750] Lustre: 269723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9837.593200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9837.605849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9837.754759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9842.212441] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9851.698421] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9856.997869] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9857.171608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9861.664754] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9861.904172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9868.258543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9868.267290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9868.278433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9870.297708] Lustre: server umount lustre-OST0000 complete [ 9873.893909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9873.906457] Lustre: Skipped 2 previous similar messages [ 9879.009270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.014250] Lustre: Skipped 1 previous similar message [ 9884.131260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9884.140503] Lustre: Skipped 1 previous similar message [ 9888.224232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9888.743657] Lustre: server umount lustre-MDT0000 complete [ 9889.261037] LustreError: 267763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9889.294692] LustreError: 267763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9892.652337] LustreError: 267792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161724 with bad export cookie 17836123595595659333 [ 9892.668934] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9893.162747] Lustre: server umount lustre-MDT0001 complete [ 9914.768054] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 9925.776572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9926.431897] LustreError: 272660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9926.521445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9926.526545] Lustre: Skipped 2 previous similar messages [ 9932.751686] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9941.708500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9941.986481] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9947.335579] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9950.373979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9956.956772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9963.908917] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9973.013125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9973.185310] Lustre: lustre-OST0001: new disk, initializing [ 9973.188406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9973.193144] Lustre: 274967:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9975.324605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9975.339321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9975.538310] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9980.547184] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 9990.435918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9994.556670] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9999.446947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9999.786496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10009.068000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10009.081738] LustreError: Skipped 1 previous similar message [10009.088282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10009.110216] Lustre: Skipped 3 previous similar messages [10009.120311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10013.332355] Lustre: server umount lustre-MDT0000 complete [10016.559971] LustreError: 273395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161848 with bad export cookie 17836123595595660215 [10016.562132] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10016.571951] LustreError: 273395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10016.743416] LustreError: 272656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10016.766185] LustreError: 272656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10017.108718] Lustre: server umount lustre-MDT0001 complete [10022.984520] Lustre: server umount lustre-OST0000 complete [10027.491063] Lustre: server umount lustre-OST0001 complete [10046.315797] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10057.738832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10058.122933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10058.131819] Lustre: Skipped 3 previous similar messages [10063.649162] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10074.300473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10080.106373] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10083.758205] Lustre: 279164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10090.675602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10098.202697] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10103.368491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10106.917687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10112.514107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10113.668982] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10121.164977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10125.209669] Lustre: 281039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10131.203180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10131.221064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10142.107050] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:39:32 (1788161972) [10158.566899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.574027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10158.578090] Lustre: Skipped 7 previous similar messages [10158.595352] Lustre: Skipped 9 previous similar messages [10172.384137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10172.585354] Lustre: server umount lustre-MDT0000 complete [10173.928102] LustreError: 278019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10173.950829] LustreError: 278019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10176.793486] LustreError: 278052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162008 with bad export cookie 17836123595595666795 [10176.799047] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10176.804196] LustreError: 278052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10191.328503] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10191.552655] Lustre: server umount lustre-MDT0001 complete [10210.272131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10210.415329] Lustre: server umount lustre-OST0000 complete [10214.436181] Lustre: server umount lustre-OST0001 complete [10222.842872] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [10240.187835] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10290.560883] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10301.235200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10301.501040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10301.532000] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10301.626536] Lustre: lustre-MDT0000: new disk, initializing [10301.772145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10306.107845] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10316.566404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10316.686262] Lustre: 285307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10316.923350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10316.933499] Lustre: Skipped 4 previous similar messages [10316.983907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10321.924212] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10335.304391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10336.087542] Lustre: 286938:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10338.025900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10344.896216] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10357.488178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10357.603822] Lustre: 287808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10359.763616] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10359.771356] Lustre: Skipped 1 previous similar message [10359.886760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10366.101288] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10377.481128] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10381.307425] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10381.316906] Lustre: Skipped 1 previous similar message [10396.533811] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:43:47 (1788162227) [10398.690118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10398.695808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10398.701386] Lustre: Skipped 3 previous similar messages [10398.705891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10398.712869] Lustre: Skipped 16 previous similar messages [10404.805789] Lustre: server umount lustre-MDT0000 complete [10408.437986] LustreError: 285298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162240 with bad export cookie 17836123595595674425 [10408.442522] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10408.449821] LustreError: 285298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10408.799857] Lustre: server umount lustre-MDT0001 complete [10421.601723] Lustre: server umount lustre-OST0000 complete [10436.110500] Lustre: server umount lustre-OST0001 complete [10444.921359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10445.368151] LustreError: 290054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10445.393819] LustreError: 290054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [10450.585334] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10459.952239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.962410] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10474.152983] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10483.184973] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10491.510159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10498.760496] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10506.621282] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10536.887091] loop1: detected capacity change from 0 to 204800000 [10537.340370] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10537.374662] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10537.411750] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10543.340823] loop1: detected capacity change from 0 to 204800000 [10550.449755] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10550.681919] LustreError: 290042:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9979: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10550.695406] LustreError: 290042:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9979 [10550.705627] LustreError: 290042:0:(mgs_handler.c:517:mgs_target_reg()) fs9979-MDT0001: Error checking target index: rc = -13 [10550.713955] LustreError: fs9979-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10550.723769] LustreError: 293799:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -13 [10550.734813] LustreError: 293799:0:(tgt_mount.c:2206:server_put_super()) no obd fs9979-MDT0001 [10550.737770] LustreError: 293799:0:(tgt_mount.c:116:server_deregister_mount()) fs9979-MDT0001 not registered [10550.807435] Lustre: server umount fs9979-MDT0001 complete [10550.812143] LustreError: 293799:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10560.797989] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.899496] Lustre: Setting parameter fs9979-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9979-MDT0001 [10560.946462] Lustre: srv-fs9979-MDT0001: No data found on store. Initialize space. [10560.951704] Lustre: Skipped 4 previous similar messages [10561.020473] Lustre: fs9979-MDT0001: new disk, initializing [10561.029586] Lustre: Skipped 3 previous similar messages [10561.174489] Lustre: srv-fs9979-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10561.181380] Lustre: 294265:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9979-MDT0001-mdtlov: can't create new llog: rc = -115 [10561.190489] LustreError: 294265:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9979-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10566.245077] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10573.232951] Lustre: server umount fs9979-MDT0001 complete [10583.030126] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10586.355217] loop1: detected capacity change from 0 to 204800000 [10592.474077] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10593.024438] Lustre: 295836:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9979-MDT0001-mdtlov: can't create new llog: rc = -115 [10593.035789] LustreError: 295836:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9979-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10597.442528] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10602.507361] Lustre: server umount fs9979-MDT0001 complete [10613.733746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10613.748451] Lustre: Skipped 3 previous similar messages [10613.757034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10613.759435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10613.767771] LustreError: Skipped 1 previous similar message [10613.782777] Lustre: Skipped 6 previous similar messages [10618.202420] Lustre: server umount lustre-MDT0000 complete [10625.864621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10626.002146] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10630.905985] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10631.663189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10640.249181] loop1: detected capacity change from 0 to 204800000 [10647.072125] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10647.555667] Lustre: 298281:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9979-MDT0001-mdtlov: can't create new llog: rc = -115 [10647.569764] LustreError: 298281:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9979-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10653.467628] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10660.197258] Lustre: server umount fs9979-MDT0001 complete [10700.257961] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10700.655806] Lustre: server umount lustre-OST0000 complete [10711.331662] Lustre: server umount lustre-MDT0000 complete [10716.485090] LustreError: 290034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162548 with bad export cookie 17836123595595677106 [10716.497829] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10716.515924] LustreError: 290034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10717.272935] Lustre: server umount lustre-MDT0001 complete [10731.588746] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [10734.387435] Key type lgssc unregistered [10734.751897] LNet: 300627:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10734.757554] LNetError: 300627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10735.782693] LNet: Removed LNI 192.168.202.150@tcp [10736.662177] Key type .llcrypt unregistered [10736.665311] Key type ._llcrypt unregistered [10763.408552] Key type ._llcrypt registered [10763.417313] Key type .llcrypt registered [10763.627818] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [10765.279366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10765.303404] alg: No test for adler32 (adler32-zlib) [10766.336633] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10766.480256] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [10768.249895] Key type lgssc registered [10769.967668] Lustre: Echo OBD driver; http://www.lustre.org/ [10778.747036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10778.794249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10780.363017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10786.479683] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10790.894218] LustreError: 301900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.922631] LustreError: 301900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10796.003720] LustreError: 301901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.538154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.786880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10802.404493] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10810.014121] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10815.908949] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10824.089311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10824.641281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10825.656867] LustreError: 303731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10830.819554] LustreError: 303732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10832.205676] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10835.982270] LustreError: 303802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10840.729527] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10844.131918] LustreError: 303733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10844.157570] LustreError: 303733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10862.628218] loop1: detected capacity change from 0 to 204800000 [10862.862686] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10862.895417] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10862.916704] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10863.129449] LustreError: 303731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10863.160750] LustreError: 303731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10869.558608] loop1: detected capacity change from 0 to 204800000 [10877.260939] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10877.488423] Lustre: fs4249-OST0000: new disk, initializing [10877.500937] Lustre: srv-fs4249-OST0000: No data found on store. Initialize space. [10877.517484] Lustre: 305640:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10877.661789] Lustre: fs4249-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10885.065131] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10894.089285] Lustre: server umount fs4249-OST0000 complete [10895.331180] LustreError: 303733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.338834] LustreError: 303733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10905.589600] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10910.264659] loop1: detected capacity change from 0 to 204800000 [10917.533207] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10917.730471] Lustre: fs4249-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10925.904351] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10926.065740] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10936.044301] loop2: detected capacity change from 0 to 204800000 [10936.367468] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10936.382668] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10936.408696] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10942.630392] loop2: detected capacity change from 0 to 204800000 [10948.835553] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10948.900988] LustreError: fs4249-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10948.921803] LustreError: fs4249-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10948.930741] LustreError: 308680:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -13 [10948.935634] LustreError: 308680:0:(tgt_mount.c:2206:server_put_super()) no obd fs4249-OST0001 [10948.939382] LustreError: 308680:0:(tgt_mount.c:116:server_deregister_mount()) fs4249-OST0001 not registered [10948.967791] Lustre: server umount fs4249-OST0001 complete [10948.971155] LustreError: 308680:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10959.724899] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10959.846973] Lustre: fs4249-OST0001: new disk, initializing [10959.851234] Lustre: srv-fs4249-OST0001: No data found on store. Initialize space. [10959.863414] Lustre: 309232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10959.956241] Lustre: fs4249-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10960.383070] LustreError: 304404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10960.404577] LustreError: 304404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [10966.015514] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [10975.087176] Lustre: server umount fs4249-OST0000 complete [10984.199148] Lustre: server umount fs4249-OST0001 complete [10997.278138] loop1: detected capacity change from 0 to 204800000 [11003.410566] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11003.623093] Lustre: fs4249-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11010.623246] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11019.101782] loop2: detected capacity change from 0 to 204800000 [11026.663610] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11033.912579] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11042.979647] Lustre: server umount fs4249-OST0000 complete [11051.520663] Lustre: server umount fs4249-OST0001 complete [11059.178153] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11064.289322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11064.299126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11064.311705] Lustre: Skipped 1 previous similar message [11064.332547] Lustre: Skipped 1 previous similar message [11069.109092] Lustre: server umount lustre-OST0000 complete [11074.529307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11074.532561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11074.542258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11074.559716] Lustre: Skipped 1 previous similar message [11079.616444] Lustre: server umount lustre-MDT0000 complete [11083.662722] LustreError: 301882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162915 with bad export cookie 7709994377149882990 [11083.663870] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11083.674957] LustreError: 301882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11083.921169] Lustre: server umount lustre-MDT0001 complete [11098.555832] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [11102.214876] Key type lgssc unregistered [11102.502543] LNet: 315157:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11102.512226] LNetError: 315157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11102.524793] LNet: Removed LNI 192.168.202.150@tcp [11103.453178] Key type .llcrypt unregistered [11103.454536] Key type ._llcrypt unregistered [11137.100684] Key type ._llcrypt registered [11137.102353] Key type .llcrypt registered [11137.195993] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [11153.859486] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [11155.765832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11155.791964] alg: No test for adler32 (adler32-zlib) [11156.822903] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11157.061159] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [11158.752449] Key type lgssc registered [11159.878216] Lustre: Echo OBD driver; http://www.lustre.org/ [11203.443196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11203.474834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.838275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11204.893491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11205.052220] Lustre: lustre-MDT0000: new disk, initializing [11205.159027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11205.194423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11209.768954] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11222.289500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.417726] Lustre: 319019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11222.484983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11222.492352] Lustre: Skipped 1 previous similar message [11222.545803] Lustre: lustre-MDT0001: new disk, initializing [11222.631477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11222.672992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11222.689188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11227.330935] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11236.875773] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11242.818559] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11249.952263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11250.225278] Lustre: lustre-OST0000: new disk, initializing [11250.232455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11250.251532] Lustre: 320986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11250.341437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11257.707319] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11259.975575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11259.998600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11260.104466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11267.644041] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11270.008853] Lustre: Mounted lustre-client [11271.233647] Lustre: Unmounted lustre-client [11280.359273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11280.379510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11280.391759] Lustre: Skipped 1 previous similar message [11285.478697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11285.496852] Lustre: Skipped 1 previous similar message [11286.604368] Lustre: server umount lustre-OST0000 complete [11294.181402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11294.194987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11294.218666] Lustre: Skipped 1 previous similar message [11294.224632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11295.715948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11297.308452] Lustre: server umount lustre-MDT0000 complete [11301.067617] LustreError: 319009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163132 with bad export cookie 15611461050037694915 [11301.072948] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11301.088341] LustreError: 319009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11301.516800] Lustre: server umount lustre-MDT0001 complete [11314.562733] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [11318.353341] Key type lgssc unregistered [11326.114693] LNet: 1 local NIs in recovery (showing 1): 192.168.202.150@tcp [11326.880699] LNet: 323148:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11326.891306] LNetError: 323148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11400.694741] LNet: Removed LNI 192.168.202.150@tcp [11401.930117] Key type .llcrypt unregistered [11401.933765] Key type ._llcrypt unregistered [11424.255665] Key type ._llcrypt registered [11424.263488] Key type .llcrypt registered [11424.442226] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [11444.888665] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [11446.994483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11447.138188] alg: No test for adler32 (adler32-zlib) [11448.129659] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11448.420961] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [11450.256301] Key type lgssc registered [11451.796078] Lustre: Echo OBD driver; http://www.lustre.org/ [11490.958603] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:02:00 (1788163320) [11497.740083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11497.786558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.066773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11499.109105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11499.197106] Lustre: lustre-MDT0000: new disk, initializing [11499.280431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11499.301631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11504.254977] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11515.177458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11515.341313] Lustre: 326388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11515.434545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11515.439627] Lustre: Skipped 1 previous similar message [11515.573402] Lustre: lustre-MDT0001: new disk, initializing [11515.716649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11515.772784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11515.779968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11520.473273] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11530.219836] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11536.099655] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11543.896145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11544.180086] Lustre: lustre-OST0000: new disk, initializing [11544.188307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11544.196980] Lustre: 328358:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11544.285691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11550.608025] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11553.356185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11553.380851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11553.527303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11560.247989] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11579.382868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11579.384385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11579.417983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11579.453642] Lustre: Skipped 1 previous similar message [11579.480209] Lustre: Skipped 1 previous similar message [11583.729367] Lustre: server umount lustre-OST0000 complete [11589.601755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11589.626884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11594.721706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11594.724385] Lustre: Skipped 2 previous similar messages [11595.039924] Lustre: server umount lustre-MDT0000 complete [11599.309167] LustreError: 326378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163431 with bad export cookie 10908648056269291587 [11599.320959] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11599.327449] LustreError: 326378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11599.903649] Lustre: server umount lustre-MDT0001 complete [11613.264973] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [11617.212889] Key type lgssc unregistered [11617.523203] LNet: 330264:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11617.537489] LNetError: 330264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11618.608327] LNet: Removed LNI 192.168.202.150@tcp [11619.568995] Key type .llcrypt unregistered [11619.576768] Key type ._llcrypt unregistered [11647.061317] Key type ._llcrypt registered [11647.062878] Key type .llcrypt registered [11647.155803] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [11648.382571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11648.433906] alg: No test for adler32 (adler32-zlib) [11649.439255] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11649.644653] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [11651.328191] Key type lgssc registered [11652.703278] Lustre: Echo OBD driver; http://www.lustre.org/ [11663.112018] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11663.161808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11664.808200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11670.437512] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11675.105901] LustreError: 331536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.139474] LustreError: 331536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11679.733289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.180662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11684.750935] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11693.568544] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11699.714567] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11707.054719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11707.377667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11714.502707] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11724.455618] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11736.783826] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11739.204734] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11745.253899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11745.259110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11745.273813] Lustre: Skipped 2 previous similar messages [11745.286452] Lustre: Skipped 2 previous similar messages [11747.913240] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [11750.386380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11750.412193] Lustre: Skipped 2 previous similar messages [11751.683817] Lustre: server umount lustre-MDT0000 complete [11752.955149] LustreError: 331532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.496737] LustreError: 331531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.544368] LustreError: 331531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11756.849488] LustreError: 331518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163588 with bad export cookie 17719717323440020925 [11756.889628] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11758.073110] LustreError: 331532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11758.123586] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.50@tcp (stopping) [11760.628104] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11763.204591] LustreError: 334101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.210023] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.50@tcp (stopping) [11763.238377] LustreError: 334101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11763.271042] Lustre: Skipped 2 previous similar messages [11771.360600] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11771.634166] Lustre: server umount lustre-MDT0001 complete [11775.813809] Lustre: server umount lustre-OST0000 complete [11787.022673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11792.400624] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11795.496630] Lustre: 335720:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.150@tcp [11798.169739] Lustre: server umount MGS complete [11807.420831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.054753] LustreError: 336292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.084208] LustreError: 336292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11808.221432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11812.898814] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11821.595232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.887968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11827.099961] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11835.092639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11835.623441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11842.189692] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [11851.384879] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11853.813229] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11861.449163] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11863.697609] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11870.184072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11870.197474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11870.211270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11870.218489] Lustre: Skipped 3 previous similar messages [11873.761751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11873.773124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11875.721090] Lustre: server umount lustre-OST0000 complete [11883.489609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11883.496963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11886.913794] Lustre: server umount lustre-MDT0000 complete [11889.122171] LustreError: 338405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.153744] LustreError: 338405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11891.540403] LustreError: 336273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163723 with bad export cookie 17719717323440021765 [11891.546208] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11891.559358] LustreError: 336273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11891.936924] Lustre: server umount lustre-MDT0001 complete [11903.970393] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [11907.121320] Key type lgssc unregistered [11907.439084] LNet: 339731:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11907.446267] LNetError: 339731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11907.464976] LNet: Removed LNI 192.168.202.150@tcp [11908.602429] Key type .llcrypt unregistered [11908.610276] Key type ._llcrypt unregistered [11933.409593] Key type ._llcrypt registered [11933.413328] Key type .llcrypt registered [11933.526425] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [11952.908497] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [11953.776754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11953.937597] alg: No test for adler32 (adler32-zlib) [11955.080488] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11955.425919] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [11957.264134] Key type lgssc registered [11958.523374] Lustre: Echo OBD driver; http://www.lustre.org/ [11998.557938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11998.612840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11999.810809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11999.862938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12000.036256] Lustre: lustre-MDT0000: new disk, initializing [12000.193321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12000.231072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12005.664661] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12017.315143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12017.473645] Lustre: 342892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12017.526787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12017.530589] Lustre: Skipped 1 previous similar message [12017.658058] Lustre: lustre-MDT0001: new disk, initializing [12017.874331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12017.918253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12017.932797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12022.252125] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12032.196586] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12037.578404] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12043.307643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12043.560557] Lustre: lustre-OST0000: new disk, initializing [12043.565228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12043.576113] Lustre: 344861:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12043.674561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12050.072058] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12050.532294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12050.553148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12050.725239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12060.946218] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12067.864423] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12068.219661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12074.974757] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12075.222569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12081.636064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12081.641230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12081.652614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12081.665220] Lustre: Skipped 1 previous similar message [12086.758642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12086.768111] Lustre: Skipped 1 previous similar message [12091.875346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12091.883790] Lustre: Skipped 1 previous similar message [12092.384183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12092.632381] Lustre: server umount lustre-OST0000 complete [12096.995107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12097.010385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12097.033270] Lustre: Skipped 2 previous similar messages [12097.052681] Lustre: Skipped 1 previous similar message [12102.114411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12102.117247] Lustre: Skipped 1 previous similar message [12111.330465] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12111.536886] Lustre: server umount lustre-MDT0000 complete [12115.260639] LustreError: 342886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163946 with bad export cookie 7837334298403518613 [12115.282659] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12115.286738] LustreError: 342886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12115.657863] Lustre: server umount lustre-MDT0001 complete [12125.774103] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:12:35 (1788163955) [12144.715635] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [12157.019550] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [12197.941141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12198.243200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12198.264857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12198.331441] Lustre: lustre-MDT0000: new disk, initializing [12198.397618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12198.435739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12203.240259] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12214.668400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12214.806523] Lustre: 349628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12214.888763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12214.891050] Lustre: Skipped 1 previous similar message [12214.999912] Lustre: lustre-MDT0001: new disk, initializing [12215.129897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12215.178913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12215.187954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12220.225073] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12230.127505] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12237.928802] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12246.187700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12246.609286] Lustre: lustre-OST0000: new disk, initializing [12246.612824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12246.627277] Lustre: 351598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12246.766827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12254.063131] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [12256.350822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12256.367448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12256.512450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12264.837855] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12302.828244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12302.835809] LustreError: Skipped 1 previous similar message [12302.846136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12302.872634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12302.875504] Lustre: Skipped 3 previous similar messages [12313.058904] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12313.307948] Lustre: server umount lustre-OST0000 complete [12317.665964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12317.675708] LustreError: Skipped 1 previous similar message [12317.677900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.688201] Lustre: Skipped 1 previous similar message [12323.252802] Lustre: server umount lustre-MDT0000 complete [12323.300908] LustreError: 349641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12323.322047] LustreError: 349641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12327.351230] LustreError: 349620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164159 with bad export cookie 7837334298403519488 [12327.356409] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12327.360826] LustreError: 349620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12327.768723] Lustre: server umount lustre-MDT0001 complete [12340.514857] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12344.394656] Key type lgssc unregistered [12344.859204] LNet: 353772:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12344.866483] LNetError: 353772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12344.891960] LNet: Removed LNI 192.168.202.150@tcp [12346.169826] Key type .llcrypt unregistered [12346.171730] Key type ._llcrypt unregistered [12445.653349] Key type ._llcrypt registered [12445.655510] Key type .llcrypt registered [12445.802446] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12446.122583] Key type .llcrypt unregistered [12446.127034] Key type ._llcrypt unregistered [12527.612505] Key type ._llcrypt registered [12527.615197] Key type .llcrypt registered [12527.730383] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12528.071325] Key type .llcrypt unregistered [12528.073233] Key type ._llcrypt unregistered [12559.541628] Key type ._llcrypt registered [12559.543873] Key type .llcrypt registered [12559.740414] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12560.068346] Key type .llcrypt unregistered [12560.073547] Key type ._llcrypt unregistered [12599.551802] Key type ._llcrypt registered [12599.560310] Key type .llcrypt registered [12599.739510] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12599.977307] Key type .llcrypt unregistered [12599.980500] Key type ._llcrypt unregistered [12686.688778] Key type ._llcrypt registered [12686.691558] Key type .llcrypt registered [12686.809398] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12687.192613] Key type .llcrypt unregistered [12687.197468] Key type ._llcrypt unregistered [12723.662312] Key type ._llcrypt registered [12723.664331] Key type .llcrypt registered [12723.792710] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12724.133400] Key type .llcrypt unregistered [12724.136429] Key type ._llcrypt unregistered [12760.490423] Key type ._llcrypt registered [12760.497899] Key type .llcrypt registered [12760.631837] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12760.960296] Key type .llcrypt unregistered [12760.962280] Key type ._llcrypt unregistered [12797.784849] Key type ._llcrypt registered [12797.788861] Key type .llcrypt registered [12797.890374] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12798.231407] Key type .llcrypt unregistered [12798.234896] Key type ._llcrypt unregistered [12834.217590] Key type ._llcrypt registered [12834.219751] Key type .llcrypt registered [12834.357175] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12834.642273] Key type .llcrypt unregistered [12834.645623] Key type ._llcrypt unregistered [12868.118879] Key type ._llcrypt registered [12868.120156] Key type .llcrypt registered [12868.228420] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12868.512542] Key type .llcrypt unregistered [12868.516893] Key type ._llcrypt unregistered [12942.933218] Key type ._llcrypt registered [12942.941988] Key type .llcrypt registered [12943.207827] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12943.773461] Key type .llcrypt unregistered [12943.775180] Key type ._llcrypt unregistered [12975.934954] Key type ._llcrypt registered [12975.937855] Key type .llcrypt registered [12976.054516] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [12976.327513] Key type .llcrypt unregistered [12976.330074] Key type ._llcrypt unregistered [13008.374540] Key type ._llcrypt registered [13008.376498] Key type .llcrypt registered [13008.481313] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [13008.748444] Key type .llcrypt unregistered [13008.749954] Key type ._llcrypt unregistered [13064.615249] Key type ._llcrypt registered [13064.618975] Key type .llcrypt registered [13064.759635] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [13081.811844] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [13082.894282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13083.072784] alg: No test for adler32 (adler32-zlib) [13084.273495] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13084.587575] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [13086.248177] Key type lgssc registered [13087.719292] Lustre: Echo OBD driver; http://www.lustre.org/ [13125.103097] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13125.169330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13126.397368] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13126.435429] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13126.522133] Lustre: tst250fs-MDT0000: new disk, initializing [13126.596856] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13126.623713] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13131.268885] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13143.286310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13143.428475] Lustre: 374128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13143.511735] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13143.514941] Lustre: Skipped 1 previous similar message [13143.611544] Lustre: tst250fs-MDT0001: new disk, initializing [13143.739499] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13143.841313] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13143.847564] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13149.551895] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13155.128654] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13162.693931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13162.964760] Lustre: tst250fs-OST0000: new disk, initializing [13162.967976] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13162.977676] Lustre: 375760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13163.066607] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13167.095819] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13167.106672] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13167.276137] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13169.430597] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13181.681901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13181.881491] Lustre: tst250fs-OST0001: new disk, initializing [13181.884808] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13181.890442] Lustre: 376631:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13182.008974] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13188.398949] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [13188.754081] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13188.763713] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13188.842115] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13194.181906] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13196.286898] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13196.292345] Lustre: Skipped 1 previous similar message [13215.717153] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13215.726902] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13215.750050] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13219.813516] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13219.823300] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13219.859374] Lustre: Skipped 1 previous similar message [13221.551363] Lustre: server umount tst250fs-MDT0000 complete [13225.250583] LustreError: 374118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165056 with bad export cookie 1469269762395426050 [13225.251217] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13225.261334] LustreError: 374118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13225.650569] Lustre: server umount tst250fs-MDT0001 complete [13240.557275] Lustre: server umount tst250fs-OST0000 complete [13255.063102] Lustre: server umount tst250fs-OST0001 complete [13273.300494] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [13282.730487] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [13317.193459] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:32:26 (1788165146) [13319.091988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13343.229986] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [13353.743129] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [13391.443678] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13091 sec ========= 04:33:41 (1788165221) [13393.733942] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:33:43 (1788165223) === [13398.164398] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:33:47 (1788165227) === [13446.988563] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [13450.190685] Key type lgssc unregistered [13450.517443] LNet: 385783:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13450.528341] LNetError: 385783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13450.545871] LNet: Removed LNI 192.168.202.150@tcp [13451.700503] Key type .llcrypt unregistered [13451.703419] Key type ._llcrypt unregistered