[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 431972625 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002263] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006728] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009090] LSM: Security Framework initializing [ 0.010033] Yama: becoming mindful. [ 0.011022] SELinux: Initializing. [ 0.012053] *** VALIDATE selinux *** [ 0.019445] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023082] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024115] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025072] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026091] *** VALIDATE tmpfs *** [ 0.028356] *** VALIDATE proc *** [ 0.029202] *** VALIDATE cgroup *** [ 0.030004] *** VALIDATE cgroup2 *** [ 0.032089] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035019] Spectre V2 : User space: Vulnerable [ 0.036004] Speculative Store Bypass: Vulnerable [ 0.039206] debug: unmapping init [mem 0xffffffff98659000-0xffffffff98660fff] [ 0.041851] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042535] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043016] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.045914] ... value mask: 0000ffffffffffff [ 0.046008] ... max period: 00007fffffffffff [ 0.047005] ... fixed-purpose events: 3 [ 0.047930] ... event mask: 000000070000000f [ 0.049196] rcu: Hierarchical SRCU implementation. [ 0.051302] smp: Bringing up secondary CPUs ... [ 0.052446] x86: Booting SMP configuration: [ 0.053014] .... node #0, CPUs: #1 #2 #3 [ 0.057664] smp: Brought up 1 node, 4 CPUs [ 0.059020] smpboot: Max logical packages: 1 [ 0.060012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.221022] node 0 deferred pages initialised in 159ms [ 0.224175] devtmpfs: initialized [ 0.225219] x86/mm: Memory block size: 128MB [ 0.227528] gcov: version magic: 0x41383552 [ 0.228685] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.231098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233293] pinctrl core: initialized pinctrl subsystem [ 0.235135] [ 0.235660] ************************************************************* [ 0.238010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239008] ** ** [ 0.241009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.243009] ** ** [ 0.245011] ** This means that this kernel is built to expose internal ** [ 0.247010] ** IOMMU data structures, which may compromise security on ** [ 0.249008] ** your system. ** [ 0.251012] ** ** [ 0.253011] ** If you see this message and you are not debugging the ** [ 0.255009] ** kernel, report this immediately to your vendor! ** [ 0.257010] ** ** [ 0.259008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260009] ************************************************************* [ 0.262514] NET: Registered protocol family 16 [ 0.264315] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.266048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.269056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.273028] cpuidle: using governor menu [ 0.274629] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.276506] PCI: Using configuration type 1 for base access [ 0.278120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.287080] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.289037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.292126] cryptd: max_cpu_qlen set to 1000 [ 0.294228] ACPI: Added _OSI(Module Device) [ 0.296023] ACPI: Added _OSI(Processor Device) [ 0.297016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.298007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.301589] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.306409] ACPI: Interpreter enabled [ 0.308068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.309008] ACPI: Using IOAPIC for interrupt routing [ 0.311082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.314328] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.322938] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.327013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.330058] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.334232] acpiphp: Slot [2] registered [ 0.335095] acpiphp: Slot [5] registered [ 0.336096] acpiphp: Slot [6] registered [ 0.338073] acpiphp: Slot [7] registered [ 0.339093] acpiphp: Slot [8] registered [ 0.340080] acpiphp: Slot [9] registered [ 0.341071] acpiphp: Slot [10] registered [ 0.342091] acpiphp: Slot [3] registered [ 0.343068] acpiphp: Slot [4] registered [ 0.345111] acpiphp: Slot [11] registered [ 0.346065] acpiphp: Slot [12] registered [ 0.347074] acpiphp: Slot [13] registered [ 0.348101] acpiphp: Slot [14] registered [ 0.350083] acpiphp: Slot [15] registered [ 0.351077] acpiphp: Slot [16] registered [ 0.352088] acpiphp: Slot [17] registered [ 0.353015] acpiphp: Slot [18] registered [ 0.353954] acpiphp: Slot [19] registered [ 0.355068] acpiphp: Slot [20] registered [ 0.356051] acpiphp: Slot [21] registered [ 0.357048] acpiphp: Slot [22] registered [ 0.358104] acpiphp: Slot [23] registered [ 0.360089] acpiphp: Slot [24] registered [ 0.361070] acpiphp: Slot [25] registered [ 0.362075] acpiphp: Slot [26] registered [ 0.363075] acpiphp: Slot [27] registered [ 0.364091] acpiphp: Slot [28] registered [ 0.365059] acpiphp: Slot [29] registered [ 0.367110] acpiphp: Slot [30] registered [ 0.368102] acpiphp: Slot [31] registered [ 0.369084] PCI host bridge to bus 0000:00 [ 0.370012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.373020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.375028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.378027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.380020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.382027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.384207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.386917] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.389302] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.399019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.404121] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.407023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.409021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.411026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.414039] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.417107] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.419057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.422865] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.428025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.440021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.447038] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.452951] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.459015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.467011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.482014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.495099] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.500016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.507014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.530014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.539705] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.545017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.550027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.565023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.573270] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.578990] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.584015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.599016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.611589] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.617011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.622015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.639923] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.646012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.651012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.672017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682454] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.685312] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.687269] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.690392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.692238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.697765] iommu: Default domain type: Passthrough [ 0.700515] SCSI subsystem initialized [ 0.702163] ACPI: bus type USB registered [ 0.704092] usbcore: registered new interface driver usbfs [ 0.706101] usbcore: registered new interface driver hub [ 0.708095] usbcore: registered new device driver usb [ 0.710167] pps_core: LinuxPPS API ver. 1 registered [ 0.711007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.714074] PTP clock support registered [ 0.716239] EDAC MC: Ver: 3.0.0 [ 0.719106] PCI: Using ACPI for IRQ routing [ 0.720785] NetLabel: Initializing [ 0.722011] NetLabel: domain hash size = 128 [ 0.724012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.726091] NetLabel: unlabeled traffic allowed by default [ 0.728163] vgaarb: loaded [ 0.730272] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.731012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.737247] clocksource: Switched to clocksource kvm-clock [ 0.831829] VFS: Disk quotas dquot_6.6.0 [ 0.833608] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.836179] *** VALIDATE ramfs *** [ 0.837378] *** VALIDATE hugetlbfs *** [ 0.839212] pnp: PnP ACPI init [ 0.841036] pnp: PnP ACPI: found 6 devices [ 0.855292] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.858819] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.860641] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.862469] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.864619] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.866717] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.869329] NET: Registered protocol family 2 [ 0.872067] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.877121] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.880813] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.885224] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.888720] TCP: Hash tables configured (established 65536 bind 65536) [ 0.891459] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.894489] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.897201] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.899842] NET: Registered protocol family 1 [ 0.902179] RPC: Registered named UNIX socket transport module. [ 0.904110] RPC: Registered udp transport module. [ 0.905663] RPC: Registered tcp transport module. [ 0.907022] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.909120] NET: Registered protocol family 44 [ 0.910480] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.912066] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.913813] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.916259] PCI: CLS 0 bytes, default 64 [ 0.918559] Unpacking initramfs... [ 2.238527] debug: unmapping init [mem 0xffff8a85bcc54000-0xffff8a85bffbffff] [ 2.241122] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.242754] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.244710] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.713901] Initialise system trusted keyrings [ 2.715577] Key type blacklist registered [ 2.717323] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.726108] zbud: loaded [ 2.728719] *** VALIDATE nfs *** [ 2.729663] *** VALIDATE nfs4 *** [ 2.730929] pstore: using deflate compression [ 2.733406] Platform Keyring initialized [ 2.832916] NET: Registered protocol family 38 [ 2.834440] Key type asymmetric registered [ 2.835649] Asymmetric key parser 'x509' registered [ 2.837215] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.839994] io scheduler mq-deadline registered [ 2.841667] io scheduler kyber registered [ 2.843287] io scheduler bfq registered [ 2.845050] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.848077] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.850404] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.852608] ACPI: Power Button [PWRF] [ 2.857026] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.863456] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.875320] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.881936] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.895357] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.922890] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.950963] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.956206] Non-volatile memory driver v1.3 [ 2.957628] Linux agpgart interface v0.103 [ 2.984712] virtio_blk virtio1: [vda] 149936 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.988372] vda: detected capacity change from 0 to 76767232 [ 3.000205] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.002711] vdb: detected capacity change from 0 to 1073741824 [ 3.013865] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.016429] vdc: detected capacity change from 0 to 2621440000 [ 3.028463] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.030662] vdd: detected capacity change from 0 to 2621440000 [ 3.043424] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.045499] vde: detected capacity change from 0 to 4294967296 [ 3.056329] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.058519] vdf: detected capacity change from 0 to 4294967296 [ 3.063583] libphy: Fixed MDIO Bus: probed [ 3.067802] usbcore: registered new interface driver usbserial_generic [ 3.070125] usbserial: USB Serial support registered for generic [ 3.072520] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.076131] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.077581] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.079241] mousedev: PS/2 mouse device common for all mice [ 3.081123] rtc_cmos 00:05: RTC can wake from S4 [ 3.083328] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.083692] rtc_cmos 00:05: registered as rtc0 [ 3.089185] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.089738] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.091829] intel_pstate: CPU model not supported [ 3.096834] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.100666] hid: raw HID events driver (C) Jiri Kosina [ 3.102885] usbcore: registered new interface driver usbhid [ 3.104571] usbhid: USB HID core driver [ 3.105957] drop_monitor: Initializing network drop monitor service [ 3.107800] Initializing XFRM netlink socket [ 3.109558] NET: Registered protocol family 10 [ 3.111924] Segment Routing with IPv6 [ 3.113221] NET: Registered protocol family 17 [ 3.115390] mpls_gso: MPLS GSO support [ 3.120137] RAS: Correctable Errors collector initialized. [ 3.122017] AVX version of gcm_enc/dec engaged. [ 3.123719] AES CTR mode by8 optimization enabled [ 3.186893] sched_clock: Marking stable (3186869525, 0)->(3934277910, -747408385) [ 3.190854] registered taskstats version 1 [ 3.192685] Loading compiled-in X.509 certificates [ 3.194062] zswap: loaded using pool lzo/zbud [ 3.213960] Key type big_key registered [ 3.225842] Key type encrypted registered [ 3.227792] ima: No TPM chip found, activating TPM-bypass! [ 3.229639] ima: Allocated hash algorithm: sha1 [ 3.230824] ima: No architecture policies found [ 3.232587] evm: Initialising EVM extended attributes: [ 3.234458] evm: security.selinux [ 3.235464] evm: security.ima [ 3.236483] evm: security.capability [ 3.237711] evm: HMAC attrs: 0x1 [ 3.240042] rtc_cmos 00:05: setting system clock to 2026-09-08 10:59:16 UTC (1788865156) [ 3.246369] debug: unmapping init [mem 0xffffffff99603000-0xffffffff997fffff] [ 3.249364] debug: unmapping init [mem 0xffffffff98382000-0xffffffff98658fff] [ 3.256198] Write protecting the kernel read-only data: 28672k [ 3.259958] debug: unmapping init [mem 0xffffffff96a03000-0xffffffff96bfffff] [ 3.263347] debug: unmapping init [mem 0xffffffff97314000-0xffffffff973fffff] [ 3.305296] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.313285] systemd[1]: Detected virtualization kvm. [ 3.315145] systemd[1]: Detected architecture x86-64. [ 3.316728] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.343502] systemd[1]: No hostname configured. [ 3.344888] systemd[1]: Set hostname to . [ 3.346766] random: systemd: uninitialized urandom read (16 bytes read) [ 3.348624] systemd[1]: Initializing machine ID from random generator. [ 3.398936] random: ln: uninitialized urandom read (6 bytes read) [ 3.475709] random: systemd: uninitialized urandom read (16 bytes read) [ 3.478182] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.482112] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.487674] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.028589] device-mapper: uevent: version 1.0.3 [ 4.030806] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] [ 4.249877] random: fast init done Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.653228] virtio_net virtio0 ens2: renamed from eth0 [ 4.725526] scsi host0: ata_piix [ 4.747984] scsi host1: ata_piix [ 4.749868] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.751533] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.053171] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.743684] random: crng init done [ 9.744666] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.021747] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.101312] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.314687] SELinux: Disabled at runtime. [ 11.378048] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.387495] systemd[1]: Detected virtualization kvm. [ 11.389463] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.881850] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.885480] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.890557] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.895352] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.901530] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.911026] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.930359] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ 12.042986] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 12.278947] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.673728] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.690790] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.915318] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.931503] EDAC sbridge: Ver: 1.1.2 [ 14.362560] Key type dns_resolver registered [ 14.656739] NFS: Registering the id_resolver key type [ 14.658118] Key type id_resolver registered [ 14.659106] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg605-server login: [ 39.009813] libcfs: loading out-of-tree module taints kernel. [ 39.033391] Key type ._llcrypt registered [ 39.034891] Key type .llcrypt registered [ 39.086608] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 47.007149] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 47.739700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 47.748266] alg: No test for adler32 (adler32-zlib) [ 48.778135] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 49.143225] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 50.792218] Key type lgssc registered [ 51.446811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.958454] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.849351] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 78.798659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 78.809324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 79.919106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 79.936588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 79.981354] Lustre: lustre-MDT0000: new disk, initializing [ 80.015322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 80.023455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 81.635130] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 87.281396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.322236] Lustre: 6491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 87.337912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 87.340449] Lustre: Skipped 1 previous similar message [ 87.383927] Lustre: lustre-MDT0001: new disk, initializing [ 87.412717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 87.423634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 87.428252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 88.895813] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 91.247684] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 94.705141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 94.799734] Lustre: lustre-OST0000: new disk, initializing [ 94.802296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 94.806336] Lustre: 8424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 94.827276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.091938] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.137502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 97.140881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 97.170607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 102.769446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 102.820760] Lustre: lustre-OST0001: new disk, initializing [ 102.822470] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 102.825731] Lustre: 9501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 102.848216] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 105.071499] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.504032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 107.508331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 107.525316] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 111.127894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 113.654808] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 115.639395] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing check_logdir /tmp/testlogs/ [ 117.325694] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing yml_node [ 118.960443] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 119.871053] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 120.760409] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 121.339843] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 07:01:14 EDT 2026 [ 127.235790] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 127.776757] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 143.329225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 143.329574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 143.335480] Lustre: Skipped 1 previous similar message [ 145.891612] Lustre: server umount lustre-MDT0000 complete [ 147.199930] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865300 with bad export cookie 988002557819529149 [ 147.202157] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 147.204748] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 148.448740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 148.449249] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 148.453815] Lustre: Skipped 3 previous similar messages [ 148.457711] Lustre: Skipped 4 previous similar messages [ 152.480647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 152.483115] Lustre: Skipped 1 previous similar message [ 153.390700] Lustre: server umount lustre-MDT0001 complete [ 160.957520] Lustre: server umount lustre-OST0000 complete [ 168.401362] Lustre: server umount lustre-OST0001 complete [ 170.736979] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 174.142350] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 188.289698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.394258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 188.413938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 188.456345] Lustre: lustre-MDT0000: new disk, initializing [ 188.499830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.508503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.103966] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 195.145702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.197573] Lustre: 15056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.329597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.897327] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 200.969051] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 202.971752] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 205.560821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.680074] Lustre: lustre-OST0000: new disk, initializing [ 205.681682] Lustre: Skipped 1 previous similar message [ 205.683924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.686622] Lustre: Skipped 2 previous similar messages [ 205.689738] Lustre: 17024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.728865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.731822] Lustre: Skipped 1 previous similar message [ 207.312466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.316533] Lustre: Skipped 1 previous similar message [ 207.319120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.363481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.017662] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 212.214261] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 213.909646] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 214.002357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 215.576938] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 215.665697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 217.577036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 217.579592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 217.580397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 217.588127] Lustre: Skipped 1 previous similar message [ 222.494930] Lustre: server umount lustre-OST0000 complete [ 226.273236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 226.276857] LustreError: Skipped 1 previous similar message [ 226.279404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 226.289698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 226.292867] Lustre: Skipped 1 previous similar message [ 230.181405] Lustre: server umount lustre-MDT0000 complete [ 231.723278] LustreError: 15850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865384 with bad export cookie 988002557819530661 [ 231.724594] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 231.729313] LustreError: 15850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 231.866932] Lustre: server umount lustre-MDT0001 complete [ 233.763564] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:03:06 (1788865386) [ 234.802593] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 235.529425] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:03:08 (1788865388) [ 238.350984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.567366] LustreError: 19428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 238.581525] LustreError: 19428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 238.620232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.244518] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 243.682643] LustreError: 19429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 243.886240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 245.910729] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 249.273334] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 251.653505] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 254.355837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.850022] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 259.836951] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 273.893290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 273.893380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.896387] LustreError: Skipped 1 previous similar message [ 273.905861] Lustre: Skipped 1 previous similar message [ 273.911647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 273.918088] Lustre: Skipped 3 previous similar messages [ 278.561245] Lustre: server umount lustre-OST0000 complete [ 280.033874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 284.129757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.134308] Lustre: Skipped 1 previous similar message [ 294.368249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 294.369450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.373533] Lustre: Skipped 6 previous similar messages [ 294.440265] Lustre: server umount lustre-MDT0000 complete [ 295.760720] LustreError: 19408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865449 with bad export cookie 988002557819531550 [ 295.764306] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.764572] LustreError: 19408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 295.895603] Lustre: server umount lustre-MDT0001 complete [ 299.879970] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 301.072838] Key type lgssc unregistered [ 301.219519] LNet: 22979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.222808] LNetError: 22979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.231526] LNet: Removed LNI 192.168.206.105@tcp [ 301.618224] Key type .llcrypt unregistered [ 301.619734] Key type ._llcrypt unregistered [ 311.023661] Key type ._llcrypt registered [ 311.024955] Key type .llcrypt registered [ 311.066750] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 311.388164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 311.396756] alg: No test for adler32 (adler32-zlib) [ 312.278061] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 312.398075] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 313.984183] Key type lgssc registered [ 314.457886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 317.845207] loop: module loaded [ 317.871690] loop0: detected capacity change from 0 to 204800000 [ 317.960440] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 317.967711] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.098298] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 320.543067] loop0: detected capacity change from 0 to 204800000 [ 320.660179] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 320.664874] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 320.670185] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 322.587802] loop0: detected capacity change from 0 to 204800000 [ 324.765854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 324.774774] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 325.866248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 325.884450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 325.923470] Lustre: lustre-MDT0000: new disk, initializing [ 325.956364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 325.968482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 327.666347] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 331.606274] loop1: detected capacity change from 0 to 204800000 [ 333.803216] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 333.923950] Lustre: lustre-OST07c6: new disk, initializing [ 333.927631] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 333.930574] Lustre: Skipped 1 previous similar message [ 333.933673] Lustre: 26140:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 333.969204] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 334.244117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 334.248608] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 334.261318] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 336.222933] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 339.572722] Lustre: 26178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 339.577560] Lustre: 26178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 339.585952] Lustre: 26178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 339.592549] Lustre: 26178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 339.598601] Lustre: 26178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 339.602325] Lustre: 26178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 361.473899] Lustre: 25124:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 361.478672] Lustre: 25124:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 361.482737] Lustre: 25124:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 361.486041] Lustre: 25124:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 361.492448] Lustre: 25124:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 361.496640] Lustre: 25124:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 365.315340] Lustre: 25125:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 365.319034] Lustre: 25125:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 365.322292] Lustre: 25125:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 365.325768] Lustre: 25125:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.329931] Lustre: 25125:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 365.333906] Lustre: 25125:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.340014] Lustre: 25125:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 365.345781] Lustre: 25125:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.350331] Lustre: 25125:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 365.354248] Lustre: 25125:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.357637] Lustre: 25125:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 365.363118] Lustre: 25125:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.447683] Lustre: 26178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 368.452431] Lustre: 26178:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 368.455662] Lustre: 26178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 368.458193] Lustre: 26178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.464879] Lustre: 26178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 368.468660] Lustre: 26178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.471362] Lustre: 26178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 368.474462] Lustre: 26178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.477801] Lustre: 26178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 368.480428] Lustre: 26178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.483105] Lustre: 26178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 368.485992] Lustre: 26178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 375.169677] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 375.172939] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 375.176009] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 375.179209] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 375.182383] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 375.184918] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 375.187483] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 375.190490] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 375.194078] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 375.197173] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 375.200371] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 375.203597] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 385.483202] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 385.488684] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 385.492544] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 385.495318] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 385.501162] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 385.505296] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 385.508734] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 385.512904] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 385.517513] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 385.521220] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 385.525633] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 385.529736] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 426.465923] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.466218] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 426.471961] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 431.586644] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 432.406019] Lustre: server umount lustre-OST07c6 complete [ 436.343970] Lustre: server umount lustre-MDT0000 complete [ 442.799465] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 444.002241] Key type lgssc unregistered [ 444.161897] LNet: 28695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.164854] LNetError: 28695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 444.172528] LNet: Removed LNI 192.168.206.105@tcp [ 444.538962] Key type .llcrypt unregistered [ 444.540735] Key type ._llcrypt unregistered [ 452.338595] Key type ._llcrypt registered [ 452.340063] Key type .llcrypt registered [ 452.388299] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 452.781357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 452.812715] alg: No test for adler32 (adler32-zlib) [ 453.683363] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 453.791639] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 455.384214] Key type lgssc registered [ 455.824934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 458.674864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 458.681045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.850689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.325105] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 464.384538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.523544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.096861] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 468.740220] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 470.842486] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 473.389479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 473.540138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.796563] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 478.595305] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 485.863806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 496.099855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.103931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.104423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.113389] Lustre: Skipped 1 previous similar message [ 497.442783] Lustre: server umount lustre-OST0000 complete [ 500.704933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 500.708359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.714978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.717532] Lustre: Skipped 1 previous similar message [ 505.306975] Lustre: server umount lustre-MDT0000 complete [ 506.338141] LustreError: 29936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.345521] LustreError: 29936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 506.696314] LustreError: 29918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865659 with bad export cookie 10674183309850301277 [ 506.700591] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.701394] LustreError: 29918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 506.831840] Lustre: server umount lustre-MDT0001 complete [ 511.192028] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 512.500047] Key type lgssc unregistered [ 512.666485] LNet: 33687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.670292] LNetError: 33687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 512.679381] LNet: Removed LNI 192.168.206.105@tcp [ 513.068689] Key type .llcrypt unregistered [ 513.070060] Key type ._llcrypt unregistered [ 521.654783] Key type ._llcrypt registered [ 521.655848] Key type .llcrypt registered [ 521.697878] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 522.148440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 522.162235] alg: No test for adler32 (adler32-zlib) [ 523.037660] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 523.143421] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 524.736267] Key type lgssc registered [ 525.242332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 528.835486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 528.845379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.077363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.729899] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 535.427209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.608841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 537.263360] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 540.129339] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 542.338481] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 544.972716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.112272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.304299] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 550.102696] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 556.392581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 566.756205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.757673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 566.757870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.760831] Lustre: Skipped 1 previous similar message [ 570.479418] Lustre: server umount lustre-OST0000 complete [ 576.481382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 576.485297] LustreError: Skipped 1 previous similar message [ 576.487121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.494046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 576.499789] Lustre: Skipped 1 previous similar message [ 582.113149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 582.115775] Lustre: Skipped 2 previous similar messages [ 586.208182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 586.278957] Lustre: server umount lustre-MDT0000 complete [ 587.233298] LustreError: 36234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.241191] LustreError: 36234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 587.765359] LustreError: 37429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865741 with bad export cookie 14505528357222244600 [ 587.769283] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.772379] LustreError: 37429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.887823] Lustre: server umount lustre-MDT0001 complete [ 591.983755] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 593.171529] Key type lgssc unregistered [ 593.320234] LNet: 38482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.323721] LNetError: 38482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.336507] LNet: Removed LNI 192.168.206.105@tcp [ 593.714084] Key type .llcrypt unregistered [ 593.716454] Key type ._llcrypt unregistered [ 603.367444] Key type ._llcrypt registered [ 603.369036] Key type .llcrypt registered [ 603.424618] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 603.786337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 603.849789] alg: No test for adler32 (adler32-zlib) [ 604.734523] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 604.832486] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 606.424211] Key type lgssc registered [ 606.854367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 609.904662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 609.911829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.101144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.747575] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 616.009363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.184832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 617.684161] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 620.472371] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 622.740551] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 625.347549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 625.494413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 627.869112] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 630.564912] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 635.751384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 640.995530] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.997131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.999619] Lustre: Skipped 1 previous similar message [ 642.326716] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.5@tcp (stopping) [ 642.330243] Lustre: Skipped 2 previous similar messages [ 645.667891] Lustre: server umount lustre-MDT0000 complete [ 646.114528] LustreError: 41217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.123361] LustreError: 41217:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 647.160530] LustreError: 41714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865800 with bad export cookie 11897078144645137609 [ 647.164471] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.167537] LustreError: 41714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 647.435368] LustreError: 39895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.446418] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.5@tcp (stopping) [ 651.232954] LustreError: 39868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.233361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.244632] Lustre: Skipped 1 previous similar message [ 651.247369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 653.365380] Lustre: server umount lustre-MDT0001 complete [ 671.510590] Lustre: server umount lustre-OST0000 complete [ 677.481742] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 678.715885] Key type lgssc unregistered [ 678.872854] LNet: 43730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.878200] LNetError: 43730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.887452] LNet: Removed LNI 192.168.206.105@tcp [ 679.261984] Key type .llcrypt unregistered [ 679.263462] Key type ._llcrypt unregistered [ 689.375612] Key type ._llcrypt registered [ 689.376846] Key type .llcrypt registered [ 689.416195] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 689.786211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.795293] alg: No test for adler32 (adler32-zlib) [ 690.668629] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 690.775951] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 692.368174] Key type lgssc registered [ 692.779522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 695.615425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 695.623583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.795322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.377894] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 701.723569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.891170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 703.495774] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 706.529881] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 708.627242] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 711.276097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 711.441193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.833778] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 716.711996] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 722.729659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 728.035345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.036896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 728.037588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 728.040434] Lustre: Skipped 1 previous similar message [ 728.048948] LustreError: Skipped 1 previous similar message [ 731.558730] Lustre: server umount lustre-OST0000 complete [ 745.086110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.091438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 745.093543] Lustre: Skipped 3 previous similar messages [ 748.194783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 748.199234] Lustre: Skipped 1 previous similar message [ 751.271281] Lustre: server umount lustre-MDT0000 complete [ 753.093825] LustreError: 45100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788865906 with bad export cookie 15954907627496910250 [ 753.099061] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.100917] LustreError: 45100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 759.306320] Lustre: server umount lustre-MDT0001 complete [ 763.775345] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 765.090296] Key type lgssc unregistered [ 765.250531] LNet: 48829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.254722] LNetError: 48829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.262549] LNet: Removed LNI 192.168.206.105@tcp [ 765.679431] Key type .llcrypt unregistered [ 765.681104] Key type ._llcrypt unregistered [ 776.488430] Key type ._llcrypt registered [ 776.490112] Key type .llcrypt registered [ 776.540528] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 776.963363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.006452] alg: No test for adler32 (adler32-zlib) [ 777.901635] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 778.014623] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 779.632286] Key type lgssc registered [ 780.257411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 784.560654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 784.570402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.794951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 787.771924] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 791.631366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.808752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 793.450261] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 796.555075] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 798.882647] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 802.107041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.281504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.926250] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 808.265544] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 811.499980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 820.900941] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 822.351180] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 833.185495] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 7h13m05s [ 833.801208] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 835.348752] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.5@tcp (stopping) [ 836.362238] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.5@tcp (stopping) [ 838.115292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 838.115370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.116739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.125858] Lustre: Skipped 1 previous similar message [ 840.837728] Lustre: server umount lustre-MDT0000 complete [ 842.325320] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 842.508987] LustreError: 52924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.519663] LustreError: 52924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 845.283079] LustreError: 50222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.297283] LustreError: 50222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 845.611306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.682216] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 845.821372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.870193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 848.084904] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 849.838702] Lustre: DEBUG MARKER: Wait for df (36560) ... [ 850.552871] Lustre: DEBUG MARKER: done [ 851.927262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 853.472858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 853.478576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.484028] Lustre: Skipped 1 previous similar message [ 853.486456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 853.489268] Lustre: Skipped 5 previous similar messages [ 856.033308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 856.037869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.177964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 862.183108] Lustre: Skipped 3 previous similar messages [ 866.272128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 866.345058] Lustre: server umount lustre-OST0000 complete [ 872.420920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.421553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 872.422841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 872.428347] Lustre: Skipped 1 previous similar message [ 872.437327] Lustre: Skipped 1 previous similar message [ 874.413883] Lustre: server umount lustre-MDT0000 complete [ 876.407862] LustreError: 52864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866029 with bad export cookie 3472099206971809938 [ 876.409090] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 876.418507] LustreError: 52864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 876.593935] Lustre: server umount lustre-MDT0001 complete [ 882.235856] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 883.756166] Key type lgssc unregistered [ 883.917842] LNet: 55153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.926302] LNetError: 55153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.935736] LNet: Removed LNI 192.168.206.105@tcp [ 884.399287] Key type .llcrypt unregistered [ 884.401050] Key type ._llcrypt unregistered [ 903.930980] Key type ._llcrypt registered [ 903.932430] Key type .llcrypt registered [ 903.992167] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 904.416891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.463889] alg: No test for adler32 (adler32-zlib) [ 905.356814] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 905.477297] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 907.080250] Key type lgssc registered [ 907.530052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 910.869612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 910.877222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.998941] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 912.010908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 912.084310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.814479] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 917.345181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.398137] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 917.416431] Lustre: 57335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 917.565260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 919.218313] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 922.122742] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 924.197396] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 926.827439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.904816] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 926.981190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.896195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 929.224448] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 931.821524] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 936.369501] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 7h14m49s [ 936.983769] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 939.686540] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 940.604484] Lustre: DEBUG MARKER: Creating a test file and stat it [ 940.618870] LustreError: 59333:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 960.797345] Lustre: lustre-MDT0000: Client a3de03e4-bf2a-4637-877a-46759966d6dd (at 192.168.206.5@tcp) reconnecting [ 980.664075] LustreError: 59333:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 981.720146] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 982.699815] Lustre: DEBUG MARKER: done [ 984.546085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 984.546779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.547569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.548584] LustreError: Skipped 1 previous similar message [ 984.557218] Lustre: Skipped 1 previous similar message [ 989.667919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.670390] Lustre: Skipped 1 previous similar message [ 990.487587] Lustre: server umount lustre-OST0000 complete [ 994.272883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 994.275206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.279731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.282021] Lustre: Skipped 1 previous similar message [ 997.979231] Lustre: server umount lustre-MDT0000 complete [ 999.203425] LustreError: 57328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866152 with bad export cookie 701722836838446859 [ 999.205954] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.206943] LustreError: 57328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 999.310618] Lustre: server umount lustre-MDT0001 complete [ 1002.983715] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1004.070117] Key type lgssc unregistered [ 1004.203470] LNet: 61658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.207097] LNetError: 61658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.215277] LNet: Removed LNI 192.168.206.105@tcp [ 1004.558895] Key type .llcrypt unregistered [ 1004.560109] Key type ._llcrypt unregistered [ 1022.396229] Key type ._llcrypt registered [ 1022.397574] Key type .llcrypt registered [ 1022.445661] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1022.801918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.853593] alg: No test for adler32 (adler32-zlib) [ 1023.729216] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1023.838404] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1025.424206] Key type lgssc registered [ 1025.841182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1027.184585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1027.196768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.319306] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1028.328817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1028.398153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.813252] Lustre: Failing over lustre-MDT0000 [ 1028.927552] Lustre: server umount lustre-MDT0000 complete [ 1031.347637] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:16:24 (1788866184) [ 1033.983755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.140279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.804134] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1039.424595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.463875] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1039.478128] Lustre: 64426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1039.590135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.482447] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1044.727055] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1047.113985] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1050.227696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1050.329390] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1050.430472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.262955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1053.338799] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1056.419073] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1059.619326] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1062.883586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.885523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.891552] Lustre: Skipped 2 previous similar messages [ 1062.894484] Lustre: Skipped 1 previous similar message [ 1067.614571] Lustre: server umount lustre-MDT0000 complete [ 1069.019612] LustreError: 64418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866222 with bad export cookie 17118113824240098551 [ 1069.022764] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1069.024217] LustreError: 64418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1069.145563] Lustre: server umount lustre-MDT0001 complete [ 1070.432931] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1072.174139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1075.543879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.264635] LustreError: 63349:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f2fb5c6f x1875762140446464/t0(0) o250->MGC192.168.206.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1079.396177] LustreError: 66276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.404820] LustreError: 66276:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1079.435185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.473127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1080.997265] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1084.276866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.381393] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.077348] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1088.899319] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1089.504087] Lustre: 63353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788866226/real 1788866226] req@000000006461caa4 x1875762140446080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788866242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1089.513054] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1090.528124] Lustre: 63351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788866227/real 1788866227] req@000000003475f889 x1875762140446208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788866243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1090.954918] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1093.984250] Lustre: 63351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788866231/real 1788866231] req@0000000054fe7c0e x1875762140446336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788866247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1093.985288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.986883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1094.000876] Lustre: Skipped 2 previous similar messages [ 1099.093468] Lustre: server umount lustre-MDT0000 complete [ 1099.233665] LustreError: 68191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.240070] LustreError: 68191:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1100.422443] LustreError: 64420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866253 with bad export cookie 17118113824240100819 [ 1100.426142] LustreError: 64420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1100.426869] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.537884] Lustre: server umount lustre-MDT0001 complete [ 1102.765291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1106.101103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.496586] LustreError: 63349:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000d34960c x1875762140465536/t0(0) o250->MGC192.168.206.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1110.607842] LustreError: 66276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.631089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.633547] Lustre: Skipped 1 previous similar message [ 1110.656568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1112.056124] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1114.955599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.053227] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.059474] Lustre: Skipped 1 previous similar message [ 1116.447105] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1118.931639] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1120.160133] Lustre: 63351:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788866257/real 1788866257] req@00000000e40a732e x1875762140465152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788866273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1120.172731] Lustre: 63351:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1120.229436] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1120.934681] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1124.832148] Lustre: 63352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788866262/real 1788866262] req@000000009254720b x1875762140465408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788866278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1124.842525] Lustre: 63352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1125.346109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1125.349400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.349973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1125.354494] Lustre: Skipped 1 previous similar message [ 1128.923554] Lustre: server umount lustre-MDT0000 complete [ 1129.441230] LustreError: 70870:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.445935] LustreError: 70870:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1130.207480] LustreError: 70016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866283 with bad export cookie 17118113824240102191 [ 1130.211800] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.212578] LustreError: 70016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1130.332642] Lustre: server umount lustre-MDT0001 complete [ 1132.777591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1134.106623] Lustre: DEBUG MARKER: files compared the same [ 1145.141931] Lustre: server umount lustre-OST0000 complete [ 1152.155519] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1153.471284] Key type lgssc unregistered [ 1153.627898] LNet: 74138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.631683] LNetError: 74138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.643493] LNet: Removed LNI 192.168.206.105@tcp [ 1154.072898] Key type .llcrypt unregistered [ 1154.074086] Key type ._llcrypt unregistered [ 1164.297127] Key type ._llcrypt registered [ 1164.298366] Key type .llcrypt registered [ 1164.340654] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1164.756758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1164.787091] alg: No test for adler32 (adler32-zlib) [ 1165.692899] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1165.796557] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1167.392239] Key type lgssc registered [ 1167.871636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1171.396722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1171.407762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.622673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.534732] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug +malloc all [ 1178.344509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.482097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1180.270473] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug +malloc all [ 1183.254892] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1185.741757] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1188.652818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1188.817256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.223243] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug +malloc all [ 1194.179587] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1197.028881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1208.289167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.289295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.293848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.298463] Lustre: Skipped 1 previous similar message [ 1208.304177] Lustre: Skipped 1 previous similar message [ 1209.554738] Lustre: server umount lustre-OST0000 complete [ 1213.215309] Lustre: server umount lustre-MDT0000 complete [ 1213.408718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.411643] LustreError: 75530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.413864] Lustre: Skipped 1 previous similar message [ 1213.424638] LustreError: 75530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1214.694773] LustreError: 75511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866367 with bad export cookie 6324944053434818525 [ 1214.696216] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1214.699083] LustreError: 75511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1214.812585] Lustre: server umount lustre-MDT0001 complete [ 1218.845383] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1220.015565] Key type lgssc unregistered [ 1220.158068] LNet: 79075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.164888] LNetError: 79075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.176902] LNet: Removed LNI 192.168.206.105@tcp [ 1220.538543] Key type .llcrypt unregistered [ 1220.540339] Key type ._llcrypt unregistered [ 1230.464766] Key type ._llcrypt registered [ 1230.466586] Key type .llcrypt registered [ 1230.520538] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1230.884245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1230.935624] alg: No test for adler32 (adler32-zlib) [ 1231.818118] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1231.931126] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1233.520158] Key type lgssc registered [ 1233.957745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1237.187494] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1237.198948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1252.640436] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1257.760294] LustreError: 80438:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.105@tcp: failed processing log, type 4: rc = -110 [ 1284.384333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.549901] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1289.225509] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1292.323296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.395798] Lustre: *** cfs_fail_loc=706, val=0*** [ 1292.570657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.599446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1294.311447] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1297.889796] LustreError: 80463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.896224] LustreError: 80463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1297.977875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.123323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1299.658680] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1302.558827] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1304.967490] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1308.130584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1308.133930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.143140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.643090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1308.649684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.658690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1309.667057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1309.669992] Lustre: Skipped 1 previous similar message [ 1312.094891] Lustre: server umount lustre-OST0000 complete [ 1313.765024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1313.769127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.774474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.905634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.912018] Lustre: Skipped 3 previous similar messages [ 1328.096181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1328.174052] Lustre: server umount lustre-MDT0000 complete [ 1329.865236] LustreError: 80445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866483 with bad export cookie 9452469761518516072 [ 1329.868499] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1329.870377] LustreError: 80445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1330.015505] Lustre: server umount lustre-MDT0001 complete [ 1334.617803] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1335.858930] Key type lgssc unregistered [ 1336.018578] LNet: 84049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.021538] LNetError: 84049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.030722] LNet: Removed LNI 192.168.206.105@tcp [ 1336.389597] Key type .llcrypt unregistered [ 1336.391353] Key type ._llcrypt unregistered [ 1347.439540] Key type ._llcrypt registered [ 1347.442464] Key type .llcrypt registered [ 1347.512363] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1347.916979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1347.929358] alg: No test for adler32 (adler32-zlib) [ 1348.825583] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1348.940422] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1350.536624] Key type lgssc registered [ 1351.004162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1354.814925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1354.826464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.788145] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1361.713878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.068881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1364.409718] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1367.522543] LustreError: 86125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.531238] LustreError: 86125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1368.765898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1368.926083] LustreError: 86123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.936311] LustreError: 86123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1368.942226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.613055] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1372.002137] LustreError: 86128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.010297] LustreError: 86128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1375.015187] LustreError: 86123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.022388] LustreError: 86123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1375.505258] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.512189] Lustre: MGS: Client 3f3c872a-a4b6-4431-9097-e092fbfc5c33 (at 0@lo) reconnecting [ 1375.524756] LustreError: 86139:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1375.646489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.690646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1377.586060] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1387.489180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1387.492746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.499845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1391.076585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1391.076635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1391.086638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.159796] Lustre: server umount lustre-OST0000 complete [ 1396.193741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.194022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1396.196340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1396.196345] Lustre: Skipped 1 previous similar message [ 1396.201804] Lustre: Skipped 1 previous similar message [ 1400.020140] Lustre: server umount lustre-MDT0000 complete [ 1401.313547] LustreError: 86123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.320438] LustreError: 86123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1401.647072] Lustre: server umount lustre-MDT0001 complete [ 1403.309698] Lustre: server umount MGS complete [ 1408.032581] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:22:40 (1788866560) [ 1420.818612] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 1424.812523] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1438.798326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.363311] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1444.757481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.866116] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1444.889740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1444.933078] Lustre: lustre-MDT0001: new disk, initializing [ 1444.974574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1444.988355] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1444.992148] Lustre: 93279:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1444.998016] LustreError: 93279:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1446.765126] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1451.874821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.989941] Lustre: lustre-OST0000: new disk, initializing [ 1451.992729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1451.996491] Lustre: 94045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1452.026823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.479885] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1456.161023] Lustre: 94479:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003dfc434b x1875762481236992/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788866625 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1458.974168] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1461.922252] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.927660] Lustre: MGS: Client f6639c9f-6c84-4c41-aae3-a00cbf16e1d7 (at 0@lo) reconnecting [ 1461.954755] Lustre: 92549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1461.965528] LustreError: 93265:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1461.970638] LustreError: 93265:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1461.984326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1462.026554] Lustre: lustre-MDT0000: new disk, initializing [ 1462.088457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.102133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1463.136573] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1463.139915] Lustre: 95114:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e222fdd8 x1875762481243136/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788866632 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1463.824200] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1470.436074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1470.439718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1472.229376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1472.234660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1472.295184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1493.986752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.987872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1493.992322] Lustre: Skipped 1 previous similar message [ 1493.995318] Lustre: Skipped 1 previous similar message [ 1499.107482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.110325] Lustre: Skipped 1 previous similar message [ 1507.296151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1507.361581] Lustre: server umount lustre-OST0000 complete [ 1509.347459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1509.347491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.351294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.357391] Lustre: Skipped 1 previous similar message [ 1509.360615] Lustre: Skipped 2 previous similar messages [ 1523.168149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1523.220595] Lustre: server umount lustre-MDT0000 complete [ 1524.705443] LustreError: 93249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.711620] LustreError: 93249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1530.995427] Lustre: server umount lustre-MDT0001 complete [ 1532.525450] Lustre: server umount MGS complete [ 1537.452141] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:24:49 (1788866689) [ 1545.131250] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1546.509331] Key type lgssc unregistered [ 1546.681813] LNet: 97609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.689344] LNetError: 97609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.705377] LNet: Removed LNI 192.168.206.105@tcp [ 1547.129558] Key type .llcrypt unregistered [ 1547.131286] Key type ._llcrypt unregistered [ 1555.673944] Key type ._llcrypt registered [ 1555.675300] Key type .llcrypt registered [ 1555.721344] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1556.117614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1556.162995] alg: No test for adler32 (adler32-zlib) [ 1557.050445] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1557.168736] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1558.768212] Key type lgssc registered [ 1559.207229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1562.318962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1562.327479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.536228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.243215] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1569.071869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.240699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1570.810565] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1573.786531] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1576.429201] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1579.372856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1579.529466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.897706] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1584.918553] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1588.722286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1593.826455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1593.828923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.836898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1593.839409] Lustre: Skipped 1 previous similar message [ 1595.680200] Lustre: server umount lustre-OST0000 complete [ 1598.945893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.946968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1598.950704] Lustre: Skipped 2 previous similar messages [ 1599.521280] Lustre: server umount lustre-MDT0000 complete [ 1601.172237] LustreError: 98785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866754 with bad export cookie 8889351551896800515 [ 1601.175901] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.178777] LustreError: 98785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1601.338956] Lustre: server umount lustre-MDT0001 complete [ 1605.528424] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1606.743894] Key type lgssc unregistered [ 1606.887767] LNet: 102330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.890583] LNetError: 102330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.899587] LNet: Removed LNI 192.168.206.105@tcp [ 1607.251239] Key type .llcrypt unregistered [ 1607.252799] Key type ._llcrypt unregistered [ 1615.107988] Key type ._llcrypt registered [ 1615.109484] Key type .llcrypt registered [ 1615.163304] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1615.582673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1615.646812] alg: No test for adler32 (adler32-zlib) [ 1616.559566] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1616.687848] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1618.289102] Key type lgssc registered [ 1618.943291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1622.198166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1622.203163] LustreError: 103381:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 1622.219619] LustreError: 103388:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 1622.222523] LustreError: 103381:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 1622.226676] LustreError: 103381:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1622.234516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.234867] LustreError: 103381:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1623.470807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.305852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.483088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1629.633985] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1632.004860] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing lsmod [ 1633.261317] LustreError: 104592:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 1633.269450] LustreError: 104593:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 1633.272940] LustreError: 104592:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 1633.276752] LustreError: 104592:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1633.281323] LustreError: 104592:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1633.283060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1633.477370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.539929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1636.835542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.837434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.842243] Lustre: Skipped 1 previous similar message [ 1641.956407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.959045] Lustre: Skipped 4 previous similar messages [ 1647.075618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.077941] Lustre: Skipped 2 previous similar messages [ 1649.120159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1649.188689] Lustre: server umount lustre-MDT0000 complete [ 1650.798077] LustreError: 103402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866804 with bad export cookie 15695584274891243436 [ 1650.802262] LustreError: 103402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1650.802424] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.128240] LustreError: 103856:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000987c3bf7 x1875762761191296/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1651.200499] Lustre: server umount lustre-MDT0001 complete [ 1663.201921] Lustre: server umount lustre-OST0000 complete [ 1666.449770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.644875] LustreError: 105660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.651625] LustreError: 105660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1666.670154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.082862] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1671.280919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.425379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1672.959277] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1675.687688] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1677.731480] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1680.451332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1680.593781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.868385] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1685.533234] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1690.858805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 1696.227458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1696.227477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.228830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1696.241311] Lustre: Skipped 2 previous similar messages [ 1698.586137] Lustre: server umount lustre-OST0000 complete [ 1701.346119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.354031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.356575] Lustre: Skipped 2 previous similar messages [ 1706.360241] Lustre: server umount lustre-MDT0000 complete [ 1706.466656] LustreError: 106387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.476946] LustreError: 106387:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1708.134307] LustreError: 105642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866861 with bad export cookie 15695584274891244367 [ 1708.137548] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1708.140148] LustreError: 105642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1708.274148] Lustre: server umount lustre-MDT0001 complete [ 1712.659538] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1713.907450] Key type lgssc unregistered [ 1714.056450] LNet: 109187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.060471] LNetError: 109187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.069789] LNet: Removed LNI 192.168.206.105@tcp [ 1714.436555] Key type .llcrypt unregistered [ 1714.437929] Key type ._llcrypt unregistered [ 1724.715194] Key type ._llcrypt registered [ 1724.716464] Key type .llcrypt registered [ 1724.757176] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1725.094927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1725.128619] alg: No test for adler32 (adler32-zlib) [ 1726.006307] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1726.121131] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1727.712291] Key type lgssc registered [ 1728.130287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1731.344942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1731.353106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.545481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.375863] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1738.177231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.344225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1739.982795] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1743.013267] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1745.324397] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1748.016027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1748.178118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.451301] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1753.612491] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1757.417503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1762.138981] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1763.470905] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1767.906779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1767.907765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.910463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.914013] Lustre: Skipped 1 previous similar message [ 1767.917589] Lustre: Skipped 1 previous similar message [ 1771.046166] Lustre: server umount lustre-OST0000 complete [ 1773.027292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.029249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.036541] Lustre: Skipped 1 previous similar message [ 1773.049729] Lustre: Skipped 1 previous similar message [ 1778.145512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1778.149642] Lustre: Skipped 1 previous similar message [ 1778.714797] Lustre: server umount lustre-MDT0000 complete [ 1780.088211] LustreError: 110557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866933 with bad export cookie 16718600312813857801 [ 1780.091107] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.093308] LustreError: 110557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1780.227136] Lustre: server umount lustre-MDT0001 complete [ 1784.263571] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1785.397671] Key type lgssc unregistered [ 1785.539125] LNet: 114207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.542454] LNetError: 114207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.552337] LNet: Removed LNI 192.168.206.105@tcp [ 1785.903666] Key type .llcrypt unregistered [ 1785.905110] Key type ._llcrypt unregistered [ 1793.346923] Key type ._llcrypt registered [ 1793.348233] Key type .llcrypt registered [ 1793.398425] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1793.739570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1793.777041] alg: No test for adler32 (adler32-zlib) [ 1794.646085] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1794.747048] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1796.344159] Key type lgssc registered [ 1796.755311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.603895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1799.612739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.792232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.335453] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1805.469449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.600832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1807.087546] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1809.663222] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1811.693865] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1814.347039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1814.469258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.662602] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1819.389254] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1826.796457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1836.000912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1836.003951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.008647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1837.026984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1837.029788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1837.030660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.194092] Lustre: server umount lustre-OST0000 complete [ 1841.633183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1841.636488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.643708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1841.647822] Lustre: Skipped 1 previous similar message [ 1846.815759] Lustre: server umount lustre-MDT0000 complete [ 1847.265313] LustreError: 115403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.274055] LustreError: 115403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1848.341376] LustreError: 115383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867001 with bad export cookie 3891898996759654143 [ 1848.344734] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1848.345698] LustreError: 115383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1848.470052] Lustre: server umount lustre-MDT0001 complete [ 1852.721388] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1853.981870] Key type lgssc unregistered [ 1854.141511] LNet: 118952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.146278] LNetError: 118952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.153479] LNet: Removed LNI 192.168.206.105@tcp [ 1854.531967] Key type .llcrypt unregistered [ 1854.533534] Key type ._llcrypt unregistered [ 1864.964478] Key type ._llcrypt registered [ 1864.965725] Key type .llcrypt registered [ 1865.011306] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1865.369899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.405025] alg: No test for adler32 (adler32-zlib) [ 1866.283237] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1866.394378] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1867.984188] Key type lgssc registered [ 1868.456735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1871.460341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1871.469603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.650415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.275883] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1877.361301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.504829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1879.261909] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1882.558141] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1884.938326] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1887.889716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.081279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.675603] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1893.801076] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1898.345553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 1907.799652] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1909.360303] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1909.363491] Lustre: Skipped 2 previous similar messages [ 1911.125727] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1911.128067] Lustre: Skipped 2 previous similar messages [ 1915.900949] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1915.912558] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1919.021682] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1919.025741] Lustre: Skipped 2 previous similar messages [ 1921.293321] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.5@tcp 192.168.0.[2,10]@tcp [ 1925.718749] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.5/24@tcp in log lustre-MDT0000 [ 1925.722505] Lustre: Skipped 2 previous similar messages [ 1927.680696] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.5/24@tcp [ 1927.684241] Lustre: Skipped 1 previous similar message [ 1933.281140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1933.285184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.288490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1933.290416] Lustre: Skipped 1 previous similar message [ 1933.296410] Lustre: Skipped 1 previous similar message [ 1936.600736] Lustre: server umount lustre-OST0000 complete [ 1938.402646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.404789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.408028] Lustre: Skipped 1 previous similar message [ 1943.522116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.524941] Lustre: Skipped 2 previous similar messages [ 1944.178240] Lustre: server umount lustre-MDT0000 complete [ 1945.885917] LustreError: 120321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867099 with bad export cookie 15130064208739611265 [ 1945.887104] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.891264] LustreError: 120321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1946.064968] Lustre: server umount lustre-MDT0001 complete [ 1950.392332] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 1951.566467] Key type lgssc unregistered [ 1951.716416] LNet: 125289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.721148] LNetError: 125289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.730470] LNet: Removed LNI 192.168.206.105@tcp [ 1952.124818] Key type .llcrypt unregistered [ 1952.126185] Key type ._llcrypt unregistered [ 1964.271208] Key type ._llcrypt registered [ 1964.272872] Key type .llcrypt registered [ 1964.317223] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 1964.730502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.777363] alg: No test for adler32 (adler32-zlib) [ 1965.653750] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1965.759269] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 1967.352173] Key type lgssc registered [ 1967.759355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1970.771275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1970.782471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.955922] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1971.959900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.967545] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.5/24@tcp [ 1973.595366] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1977.065371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.213153] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1977.224206] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.5/24@tcp [ 1977.244590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1978.812605] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1981.604813] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1983.663725] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1986.275908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1986.450990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.636580] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 1991.302971] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.736819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 2008.034001] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2008.036387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.037939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2008.040367] Lustre: Skipped 1 previous similar message [ 2011.031310] Lustre: server umount lustre-OST0000 complete [ 2013.154369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2013.155456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.157899] LustreError: Skipped 1 previous similar message [ 2013.158695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.163620] Lustre: Skipped 1 previous similar message [ 2013.165504] Lustre: Skipped 1 previous similar message [ 2018.273559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.276606] Lustre: Skipped 1 previous similar message [ 2018.546752] Lustre: server umount lustre-MDT0000 complete [ 2020.013223] LustreError: 126805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867173 with bad export cookie 8986012627493889665 [ 2020.016098] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2020.021461] LustreError: 126805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2020.175334] Lustre: server umount lustre-MDT0001 complete [ 2024.317623] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2025.607805] Key type lgssc unregistered [ 2025.765699] LNet: 130427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.768324] LNetError: 130427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.776506] LNet: Removed LNI 192.168.206.105@tcp [ 2026.143915] Key type .llcrypt unregistered [ 2026.145244] Key type ._llcrypt unregistered [ 2037.066678] Key type ._llcrypt registered [ 2037.068150] Key type .llcrypt registered [ 2037.121970] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 2044.214659] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2044.745803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2044.771402] alg: No test for adler32 (adler32-zlib) [ 2045.664070] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 2045.775371] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2047.369498] Key type lgssc registered [ 2047.815850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2061.198611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2061.206335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.295699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2062.308134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2062.348969] Lustre: lustre-MDT0000: new disk, initializing [ 2062.376722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.385087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2064.068857] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2068.722187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.774416] Lustre: 133793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2068.806571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2068.809729] Lustre: Skipped 1 previous similar message [ 2068.848280] Lustre: lustre-MDT0001: new disk, initializing [ 2068.892510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2068.912084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2068.917185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2070.491726] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2074.661253] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2076.856367] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2079.523643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2079.645033] Lustre: lustre-OST0000: new disk, initializing [ 2079.647516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2079.651229] Lustre: 135760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2079.692262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.034118] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2085.901886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2085.912067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2085.959045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2086.277301] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2088.171787] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2088.264940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2089.982797] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2090.075619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2090.977485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2090.978524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.980667] LustreError: Skipped 1 previous similar message [ 2090.985984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2090.990096] Lustre: Skipped 1 previous similar message [ 2092.839927] Lustre: server umount lustre-OST0000 complete [ 2094.562203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2094.565664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.578669] Lustre: Skipped 1 previous similar message [ 2094.582762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.097379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.098062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.511605] Lustre: server umount lustre-MDT0000 complete [ 2102.124866] LustreError: 134590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867255 with bad export cookie 997060829332998390 [ 2102.128735] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.130232] LustreError: 134590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2102.265776] Lustre: server umount lustre-MDT0001 complete [ 2105.834477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.075239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.709489] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2111.111381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.269590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.916539] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2115.764691] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2118.114313] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2121.254108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2121.428214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.061429] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2127.458228] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2129.322136] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2132.493623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2134.333970] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2134.427860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2139.250669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2139.327447] Lustre: lustre-OST0001: new disk, initializing [ 2139.329855] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2139.333982] Lustre: 141367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2139.382571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.723328] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2145.718084] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2146.819872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2146.823805] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2146.862537] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2147.885191] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2149.765793] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2149.873529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2152.030135] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d10c8176000.ost_server_uuid 50 [ 2152.679117] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d10c8176000.ost_server_uuid in FULL state after 0 sec [ 2157.029474] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2157.029819] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.032078] LustreError: Skipped 1 previous similar message [ 2157.039976] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2157.046151] Lustre: Skipped 2 previous similar messages [ 2160.976290] Lustre: server umount lustre-OST0001 complete [ 2162.149672] LustreError: 139794:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.158757] LustreError: 139794:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2162.657815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2162.662544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.669681] Lustre: Skipped 1 previous similar message [ 2162.671718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2167.269897] LustreError: 139939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.279384] LustreError: 139939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2168.603373] Lustre: server umount lustre-OST0000 complete [ 2172.386695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.391157] Lustre: Skipped 1 previous similar message [ 2172.393511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.395950] Lustre: Skipped 4 previous similar messages [ 2176.349911] Lustre: server umount lustre-MDT0000 complete [ 2177.505398] LustreError: 137974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.513828] LustreError: 137974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2177.827384] LustreError: 137955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867331 with bad export cookie 997060829332999258 [ 2177.830879] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.832099] LustreError: 137955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2177.962608] Lustre: server umount lustre-MDT0001 complete [ 2185.074565] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2186.231852] Key type lgssc unregistered [ 2186.353690] LNet: 144378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.357133] LNetError: 144378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.363716] LNet: Removed LNI 192.168.206.105@tcp [ 2186.699121] Key type .llcrypt unregistered [ 2186.700728] Key type ._llcrypt unregistered [ 2202.023814] Key type ._llcrypt registered [ 2202.025606] Key type .llcrypt registered [ 2202.074993] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 2209.286537] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2209.728531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.747353] alg: No test for adler32 (adler32-zlib) [ 2210.620713] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 2210.717929] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2212.304224] Key type lgssc registered [ 2212.766102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2228.127510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2228.140001] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.237746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2229.258256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2229.300586] Lustre: lustre-MDT0000: new disk, initializing [ 2229.328981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.335953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2231.083842] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2236.286896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.339927] Lustre: 148498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2236.370656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2236.373386] Lustre: Skipped 1 previous similar message [ 2236.414424] Lustre: lustre-MDT0001: new disk, initializing [ 2236.454214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2236.468473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2236.473099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2238.138975] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2242.635322] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2244.756979] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2247.584431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2247.693647] Lustre: lustre-OST0000: new disk, initializing [ 2247.695965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2247.699455] Lustre: 150468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2247.731095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.112763] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2254.357245] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2255.360328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2255.365188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2255.404660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2261.822759] Lustre: Failing over lustre-OST0000 [ 2261.932367] Lustre: server umount lustre-OST0000 complete [ 2275.266049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2275.385909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.390625] Lustre: Skipped 1 previous similar message [ 2275.393343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2275.400067] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2276.117052] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2277.493062] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2278.840512] Lustre: Failing over lustre-MDT0000 [ 2278.948543] Lustre: server umount lustre-MDT0000 complete [ 2280.417732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.422983] Lustre: Skipped 2 previous similar messages [ 2281.440193] Lustre: 146728:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788867418/real 1788867418] req@00000000df456b2c x1875763384053504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788867434 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2281.450642] Lustre: 146728:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2285.538187] LustreError: 148534:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.543828] LustreError: 148534:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2286.346503] LustreError: 148505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.561179] Lustre: 146727:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788867423/real 1788867423] req@00000000aa28d31a x1875763384055296/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788867439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2290.658297] LustreError: 148510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.677041] LustreError: 148510:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2293.082172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.147367] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.152759] Lustre: 148502:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2293.249336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2293.279240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.302837] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2295.323486] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2296.587260] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2298.345149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2298.356933] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2298.366280] Lustre: Skipped 1 previous similar message [ 2298.386887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2303.460632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2303.464500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.470135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2303.473259] Lustre: Skipped 1 previous similar message [ 2305.754693] Lustre: server umount lustre-OST0000 complete [ 2308.578814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.579515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2308.583894] Lustre: Skipped 1 previous similar message [ 2308.586792] LustreError: Skipped 1 previous similar message [ 2308.589250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.593109] Lustre: Skipped 1 previous similar message [ 2313.371636] Lustre: server umount lustre-MDT0000 complete [ 2313.701690] LustreError: 148505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.711474] LustreError: 148505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2314.804292] LustreError: 148492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867468 with bad export cookie 12937550548999146297 [ 2314.807188] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.809763] LustreError: 148492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2314.952842] Lustre: server umount lustre-MDT0001 complete [ 2318.805960] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2320.029412] Key type lgssc unregistered [ 2320.193742] LNet: 154112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.197417] LNetError: 154112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.208438] LNet: Removed LNI 192.168.206.105@tcp [ 2320.587094] Key type .llcrypt unregistered [ 2320.588807] Key type ._llcrypt unregistered [ 2341.611086] Key type ._llcrypt registered [ 2341.612640] Key type .llcrypt registered [ 2341.653596] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2342.009298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.060103] alg: No test for adler32 (adler32-zlib) [ 2342.922098] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 2343.018163] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2344.608348] Key type lgssc registered [ 2345.025598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2348.096378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2348.105116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.306314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.950274] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2355.091916] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2357.138571] LustreError: 155653:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2357.143072] LustreError: 155653:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2357.146814] LustreError: 155653:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2357.269972] Lustre: server umount lustre-MDT0000 complete [ 2362.589330] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 2366.164097] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2381.850333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.962516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2381.977417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2382.018286] Lustre: lustre-MDT0000: new disk, initializing [ 2382.051573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.062464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2383.879257] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2389.050579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.103293] Lustre: 158952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2389.138334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2389.141962] Lustre: Skipped 1 previous similar message [ 2389.183465] Lustre: lustre-MDT0001: new disk, initializing [ 2389.234245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2389.251707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2389.256331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2391.428928] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2396.395172] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2399.122330] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2402.604406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2402.731736] Lustre: lustre-OST0000: new disk, initializing [ 2402.738068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2402.746476] Lustre: 160918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2402.792783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.778890] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2408.459970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2408.466917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2408.522983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2410.697649] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2414.231808] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2415.013178] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:39:27 (1788867567) [ 2526.179278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.180653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2526.187018] Lustre: Skipped 2 previous similar messages [ 2526.191752] Lustre: Skipped 1 previous similar message [ 2529.667789] Lustre: server umount lustre-MDT0000 complete [ 2531.297358] LustreError: 161638:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.303576] LustreError: 161638:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2531.446578] LustreError: 159746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867684 with bad export cookie 294839823445757023 [ 2531.447269] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2531.451883] LustreError: 159746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2536.421490] LustreError: 162680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.421843] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.438572] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2536.441433] Lustre: Skipped 1 previous similar message [ 2537.685605] Lustre: server umount lustre-MDT0001 complete [ 2545.634537] Lustre: server umount lustre-OST0000 complete [ 2550.070894] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 2554.621104] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2571.046794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.138091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2571.152264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2571.193074] Lustre: lustre-MDT0000: new disk, initializing [ 2571.219827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.227631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2572.687253] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2577.298965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.349135] Lustre: 165557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2577.378216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2577.381104] Lustre: Skipped 1 previous similar message [ 2577.422860] Lustre: lustre-MDT0001: new disk, initializing [ 2577.483211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2577.487729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2579.094105] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2583.683735] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2586.439222] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2589.811104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.956961] Lustre: lustre-OST0000: new disk, initializing [ 2589.961105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2589.966152] Lustre: 167522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2590.016465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.020881] Lustre: Skipped 1 previous similar message [ 2592.989641] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2598.146372] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2598.950406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2598.955088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2599.015576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2600.495724] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2600.601440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2602.880539] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2602.993327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2604.003599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.007038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.009648] LustreError: Skipped 1 previous similar message [ 2604.011413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.016185] Lustre: Skipped 1 previous similar message [ 2606.128358] Lustre: server umount lustre-OST0000 complete [ 2609.121984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.123593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.129572] Lustre: Skipped 1 previous similar message [ 2609.134363] Lustre: Skipped 2 previous similar messages [ 2614.245614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.249714] Lustre: Skipped 1 previous similar message [ 2614.368295] Lustre: server umount lustre-MDT0000 complete [ 2616.365986] LustreError: 165547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867769 with bad export cookie 294839823445915839 [ 2616.366612] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.371428] LustreError: 165547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2616.545523] Lustre: server umount lustre-MDT0001 complete [ 2620.898821] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:42:53 (1788867773) [ 2628.792825] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 2633.162347] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2651.201443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.332572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2651.349648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2651.392814] Lustre: lustre-MDT0000: new disk, initializing [ 2651.446757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.461192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2653.431555] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2659.615325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.675549] Lustre: 172269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2659.829842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2662.550522] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2668.076675] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2671.019698] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2674.548272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2674.700386] Lustre: 174243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2677.861855] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2680.856698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2680.934848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2682.977739] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2691.044163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2691.046091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.050019] LustreError: Skipped 1 previous similar message [ 2691.051361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2691.061736] Lustre: Skipped 1 previous similar message [ 2691.070586] Lustre: Skipped 1 previous similar message [ 2692.383919] Lustre: server umount lustre-OST0000 complete [ 2695.649470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2700.357867] Lustre: server umount lustre-MDT0000 complete [ 2701.281669] LustreError: 172278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.290238] LustreError: 172278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2702.197969] LustreError: 172263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867855 with bad export cookie 294839823445916707 [ 2702.202575] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2702.205658] LustreError: 172263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2702.359035] Lustre: server umount lustre-MDT0001 complete [ 2706.461549] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:44:19 (1788867859) [ 2715.073075] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 2720.087891] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2738.812043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.944111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2738.967726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2738.970683] Lustre: Skipped 3 previous similar messages [ 2739.023917] Lustre: lustre-MDT0000: new disk, initializing [ 2739.026112] Lustre: Skipped 2 previous similar messages [ 2739.065721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.070443] Lustre: Skipped 2 previous similar messages [ 2739.077963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2739.083993] Lustre: Skipped 2 previous similar messages [ 2741.649888] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2748.281206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.368766] Lustre: 178561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2748.591205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2751.951230] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2758.163679] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2761.213117] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2765.163600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2765.334473] Lustre: 180534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2769.124688] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2772.056286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2774.458341] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2782.178985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2782.179764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.182582] LustreError: Skipped 1 previous similar message [ 2782.183120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2782.188770] Lustre: Skipped 3 previous similar messages [ 2782.190700] Lustre: Skipped 3 previous similar messages [ 2783.548475] Lustre: server umount lustre-OST0000 complete [ 2791.669558] Lustre: server umount lustre-MDT0000 complete [ 2792.417391] LustreError: 179932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.426437] LustreError: 179932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2793.660384] LustreError: 178553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867946 with bad export cookie 294839823445917778 [ 2793.662808] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.665302] LustreError: 178553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2793.826538] Lustre: server umount lustre-MDT0001 complete [ 2798.560975] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2799.880154] Key type lgssc unregistered [ 2800.040357] LNet: 182386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2800.047537] LNetError: 182386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2800.063833] LNet: Removed LNI 192.168.206.105@tcp [ 2800.497361] Key type .llcrypt unregistered [ 2800.499148] Key type ._llcrypt unregistered [ 2812.642530] Key type ._llcrypt registered [ 2812.644618] Key type .llcrypt registered [ 2812.721883] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2813.301563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.321461] alg: No test for adler32 (adler32-zlib) [ 2814.254640] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 2814.474847] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2816.088201] Key type lgssc registered [ 2816.663475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2820.903281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2820.916606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.180746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.303240] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2829.042140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.239605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2831.535190] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2834.950994] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2837.337667] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2840.511468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.710524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.513697] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2846.889664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2846.970237] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2867.681264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2867.686462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.693864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2869.731208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2869.734281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2869.737100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.656541] Lustre: server umount lustre-OST0000 complete [ 2874.850644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.851733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2874.855971] Lustre: Skipped 1 previous similar message [ 2874.861664] Lustre: Skipped 2 previous similar messages [ 2879.525812] Lustre: server umount lustre-MDT0000 complete [ 2879.969766] LustreError: 184504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.978628] LustreError: 184504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2881.141196] LustreError: 183757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868034 with bad export cookie 12251463482300901017 [ 2881.145923] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2881.146605] LustreError: 183757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2881.297416] Lustre: server umount lustre-MDT0001 complete [ 2885.883285] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 2887.206873] Key type lgssc unregistered [ 2887.354511] LNet: 187324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2887.357318] LNetError: 187324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2887.365340] LNet: Removed LNI 192.168.206.105@tcp [ 2887.753560] Key type .llcrypt unregistered [ 2887.755565] Key type ._llcrypt unregistered [ 2898.664211] Key type ._llcrypt registered [ 2898.665844] Key type .llcrypt registered [ 2898.714635] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 2899.186891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2899.235820] alg: No test for adler32 (adler32-zlib) [ 2900.130069] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 2900.245817] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 2901.848205] Key type lgssc registered [ 2902.354540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2906.046360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2906.060164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.266574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.979560] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2912.835666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.012989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2914.959710] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2918.307269] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2920.762415] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2923.505707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2923.682576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.043130] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2929.045808] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2931.882228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2943.461246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2943.463986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.471551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.474557] Lustre: Skipped 1 previous similar message [ 2944.915869] Lustre: server umount lustre-OST0000 complete [ 2947.878278] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d10c6823800.ost_server_uuid 50 [ 2964.256394] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2978.274026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.275380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2978.285669] Lustre: Skipped 2 previous similar messages [ 2978.291103] Lustre: Skipped 1 previous similar message [ 2981.177599] Lustre: server umount lustre-MDT0000 complete [ 2982.984413] LustreError: 188694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868136 with bad export cookie 5294215020404737697 [ 2982.987375] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2982.990822] LustreError: 188694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2983.136365] Lustre: server umount lustre-MDT0001 complete [ 2986.948176] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:48:59 (1788868139) [ 2989.898518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.108136] LustreError: 192645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.123326] LustreError: 192645:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2990.149483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.007983] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 2995.170241] LustreError: 192646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.117614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.302759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2998.190333] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3001.510153] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3004.063919] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3007.167052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3007.336795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3009.942314] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3013.362558] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3015.532519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3016.459496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3016.552653] Lustre: lustre-OST0001: new disk, initializing [ 3016.555179] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3016.558816] Lustre: 195485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3019.064602] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3021.331705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3021.336147] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3021.379454] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3023.789409] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3025.890465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3025.894541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.901929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3027.425875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 3027.429283] LustreError: Skipped 1 previous similar message [ 3029.275524] Lustre: lustre-OST0000: Not available for connect from 192.168.206.5@tcp (stopping) [ 3029.279081] Lustre: Skipped 2 previous similar messages [ 3031.297128] Lustre: server umount lustre-OST0000 complete [ 3032.545732] LustreError: 196007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.553344] LustreError: 196007:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3033.722331] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3033.817160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3035.672418] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3035.771428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3037.667084] LustreError: 194466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.672992] LustreError: 194466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3042.786607] LustreError: 194466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.794973] LustreError: 194466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3047.908372] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.911598] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3047.920536] Lustre: Skipped 2 previous similar messages [ 3053.028097] LustreError: 194467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.040285] LustreError: 194467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3053.170978] Lustre: server umount lustre-OST0001 complete [ 3057.637083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3057.641400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.650289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3057.653145] Lustre: Skipped 3 previous similar messages [ 3061.241297] Lustre: server umount lustre-MDT0000 complete [ 3062.972798] LustreError: 192627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868216 with bad export cookie 5294215020404738754 [ 3062.976303] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.981805] LustreError: 192627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3062.990909] LustreError: 193374:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3062.998226] LustreError: 193374:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3063.129830] Lustre: server umount lustre-MDT0001 complete [ 3073.659345] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:50:26 (1788868226) [ 3076.543245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.675931] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3076.695693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3076.794974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.797365] Lustre: Skipped 1 previous similar message [ 3078.497948] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3082.336896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.397176] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3082.419569] Lustre: 198916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3084.196778] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3087.423066] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3089.768546] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3092.600750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3092.691512] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3095.403463] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3098.533677] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3101.382881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.424834] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3101.602136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3101.612743] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3104.053758] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3107.405709] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3111.907201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3111.910338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.913548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.915941] Lustre: Skipped 2 previous similar messages [ 3111.920891] Lustre: Skipped 2 previous similar messages [ 3114.691784] Lustre: server umount lustre-OST0000 complete [ 3117.026877] LustreError: 200766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3117.038112] LustreError: 200766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3127.273251] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.278332] LustreError: Skipped 2 previous similar messages [ 3132.602991] Lustre: server umount lustre-OST0001 complete [ 3137.507691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.518675] Lustre: Skipped 3 previous similar messages [ 3140.840259] Lustre: server umount lustre-MDT0000 complete [ 3142.804123] LustreError: 198909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868296 with bad export cookie 5294215020404740140 [ 3142.804637] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.812146] LustreError: 198909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3143.000458] Lustre: server umount lustre-MDT0001 complete [ 3154.136937] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:51:46 (1788868306) [ 3161.914620] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 3166.145278] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3181.927804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.034306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3182.052163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3182.095262] Lustre: lustre-MDT0000: new disk, initializing [ 3182.126712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.130895] Lustre: Skipped 3 previous similar messages [ 3182.136891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3183.877583] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3189.044057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.100373] Lustre: 206979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3189.128348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3189.130824] Lustre: Skipped 1 previous similar message [ 3189.170723] Lustre: lustre-MDT0001: new disk, initializing [ 3189.228265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3189.235430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3191.063831] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3195.775407] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3198.557650] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3202.188703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3202.320472] Lustre: lustre-OST0000: new disk, initializing [ 3202.323725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3202.329262] Lustre: 208948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3205.725575] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3209.221410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3209.233691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3209.288101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3211.003476] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3213.053538] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3213.155422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3215.159845] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3215.259233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3219.425970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.427353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.429687] LustreError: Skipped 1 previous similar message [ 3219.437220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3219.440269] Lustre: Skipped 9 previous similar messages [ 3222.226788] Lustre: server umount lustre-OST0000 complete [ 3238.368321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.452948] Lustre: server umount lustre-MDT0000 complete [ 3238.884238] LustreError: 206991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.895800] LustreError: 206991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 3240.196279] LustreError: 206970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868393 with bad export cookie 5294215020404741827 [ 3240.197662] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.204993] LustreError: 206970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3240.354728] Lustre: server umount lustre-MDT0001 complete [ 3244.230851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.264743] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3250.044566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.233480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3250.236537] Lustre: Skipped 3 previous similar messages [ 3252.084017] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3255.274298] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3257.771391] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3261.216694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.538813] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3268.268945] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3270.728457] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3270.849393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3273.194672] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3276.387898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3279.841161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3283.359710] Lustre: server umount lustre-OST0000 complete [ 3285.904874] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3286.001403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3287.803528] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3312.398728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.162304] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3318.619560] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3319.310675] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3321.639686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.645299] Lustre: Skipped 9 previous similar messages [ 3326.760525] Lustre: server umount lustre-OST0000 complete [ 3328.479351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.485567] Lustre: Skipped 5 previous similar messages [ 3334.661296] Lustre: server umount lustre-MDT0000 complete [ 3336.618396] LustreError: 211143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868489 with bad export cookie 5294215020404742709 [ 3336.622395] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.625990] LustreError: 211143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3336.795624] Lustre: server umount lustre-MDT0001 complete [ 3340.584163] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:54:53 (1788868493) [ 3344.094503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.327094] LustreError: 217154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3344.336228] LustreError: 217154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3346.415405] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3350.621340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.868109] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3356.174809] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3358.904923] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3362.190304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.166333] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3368.387172] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3370.295242] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3370.389221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3372.203472] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3377.437198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 5 sec [ 3380.049094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3380.128632] Lustre: lustre-OST0001: new disk, initializing [ 3380.131048] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3380.135130] Lustre: 220556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3380.171418] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.174508] Lustre: Skipped 5 previous similar messages [ 3382.557705] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3387.085299] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3387.413748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3387.418985] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3387.468578] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3388.878453] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3388.978051] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3390.703482] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3390.799547] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3392.481628] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3392.487446] LustreError: Skipped 2 previous similar messages [ 3393.671839] Lustre: server umount lustre-OST0001 complete [ 3396.441571] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3396.535377] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3398.532217] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3398.644663] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3423.162035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3426.277814] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3428.328877] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3428.336395] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3428.336680] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3428.348597] Lustre: Skipped 1 previous similar message [ 3429.854800] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3430.548123] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3437.517891] Lustre: server umount lustre-OST0001 complete [ 3451.874218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3451.876438] Lustre: Skipped 9 previous similar messages [ 3456.736897] Lustre: server umount lustre-OST0000 complete [ 3458.530834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.536467] Lustre: Skipped 7 previous similar messages [ 3464.551885] Lustre: server umount lustre-MDT0000 complete [ 3466.074846] LustreError: 220555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868619 with bad export cookie 5294215020404743661 [ 3466.077502] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3466.081522] LustreError: 220555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3472.289353] Lustre: server umount lustre-MDT0001 complete [ 3482.259783] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:57:14 (1788868634) [ 3485.315632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.470377] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3485.486349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3487.448316] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3490.995584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.049899] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3491.066339] Lustre: 225789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3492.896909] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3495.814576] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3498.020597] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3500.767440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3500.872764] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3503.278957] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3506.054487] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3510.457730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3510.499451] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3513.092294] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3516.146474] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3516.740337] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3518.316321] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3518.395959] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3519.923202] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3520.006591] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3522.125876] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d10c787a000.ost_server_uuid 50 [ 3522.748189] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d10c787a000.ost_server_uuid in FULL state after 0 sec [ 3523.177448] Lustre: Permanently deactivating lustre-OST0001 [ 3523.184452] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3524.309717] Lustre: Permanently reactivating lustre-OST0001 [ 3527.652065] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3527.655357] LustreError: Skipped 5 previous similar messages [ 3532.034238] Lustre: server umount lustre-OST0001 complete [ 3532.772549] LustreError: 227631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.779728] LustreError: 227631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 27 previous similar messages [ 3539.744107] Lustre: server umount lustre-OST0000 complete [ 3549.503641] Lustre: server umount lustre-MDT0000 complete [ 3551.245747] LustreError: 225782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868704 with bad export cookie 5294215020404744928 [ 3551.250574] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3551.253870] LustreError: 225782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3551.417757] Lustre: server umount lustre-MDT0001 complete [ 3561.914111] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:58:34 (1788868714) [ 3565.217039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.381544] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3565.397916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3565.404569] Lustre: Skipped 5 previous similar messages [ 3567.232384] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3570.826249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.891398] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3570.906649] Lustre: 232472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3572.714939] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3575.755850] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3578.090258] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3581.195975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3581.320858] Lustre: Permanently deactivating lustre-OST0000 [ 3581.327223] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3583.734634] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3584.032632] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3587.189983] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3590.028397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3591.149392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3592.767571] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3596.025468] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3596.730698] Lustre: Permanently reactivating lustre-OST0000 [ 3600.076353] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3600.080805] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3600.085657] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3600.105180] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3600.114041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3613.269865] Lustre: server umount lustre-OST0001 complete [ 3621.155483] Lustre: server umount lustre-OST0000 complete [ 3629.114654] Lustre: server umount lustre-MDT0000 complete [ 3630.824294] LustreError: 235348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868784 with bad export cookie 5294215020404746573 [ 3630.827246] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3630.830667] LustreError: 235348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3630.963593] Lustre: server umount lustre-MDT0001 complete [ 3635.873833] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 3637.258137] Key type lgssc unregistered [ 3637.420654] LNet: 237323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3637.424718] LNetError: 237323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3637.437760] LNet: Removed LNI 192.168.206.105@tcp [ 3637.864322] Key type .llcrypt unregistered [ 3637.866063] Key type ._llcrypt unregistered [ 3648.120941] Key type ._llcrypt registered [ 3648.122387] Key type .llcrypt registered [ 3648.173649] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3648.567574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3648.617857] alg: No test for adler32 (adler32-zlib) [ 3649.499679] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 3649.618813] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 3651.216612] Key type lgssc registered [ 3651.684406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3655.940608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3655.949549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.179291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.964862] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3662.583989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3662.645175] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3662.661854] Lustre: 238751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3662.669427] Lustre: Permanently deactivating lustre-MDT0001 [ 3662.678340] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3662.845131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3663.326280] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3663.333171] LustreError: 238794:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3663.339205] LustreError: 238794:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3663.344568] LustreError: 238794:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3664.593882] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3667.853207] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3670.196828] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3673.367728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.555637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3676.083284] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3679.413612] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3681.768788] LustreError: 240603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3681.773708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3681.778676] LustreError: 240603:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3682.595320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3682.701871] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3685.303887] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3687.857235] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3687.860357] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3687.861782] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3688.541426] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3691.362386] Lustre: Permanently reactivating lustre-MDT0001 [ 3691.369701] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3691.372698] Lustre: Skipped 2 previous similar messages [ 3698.454398] Lustre: lustre-MDT0001: Client f1228452-7baa-472a-95ba-7d505de500dc (at 192.168.206.5@tcp) reconnecting [ 3701.062053] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3701.714670] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3704.030173] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3704.117734] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3705.925285] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3706.014504] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3707.221447] Lustre: Permanently deactivating lustre-MDT0001 [ 3707.230908] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3707.233412] Lustre: Skipped 2 previous similar messages [ 3717.643367] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3724.257243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3724.264489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.272986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3724.726880] Lustre: server umount lustre-MDT0000 complete [ 3724.770279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.777706] LustreError: 238759:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.779658] Lustre: Skipped 1 previous similar message [ 3724.792625] LustreError: 238759:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3726.626097] LustreError: 243243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868879 with bad export cookie 17992477544445107321 [ 3726.631526] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3726.633906] LustreError: 243243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3726.789620] Lustre: server umount lustre-MDT0001 complete [ 3728.650246] Lustre: server umount lustre-OST0000 complete [ 3734.060200] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:01:26 (1788868886) [ 3738.826112] Lustre: server umount lustre-OST0001 complete [ 3741.484114] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 3745.466472] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3761.357438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.453541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3761.459225] Lustre: Skipped 2 previous similar messages [ 3761.470831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3761.511108] Lustre: lustre-MDT0000: new disk, initializing [ 3761.554446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.562665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3763.272662] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3768.445984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.506963] Lustre: 247115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3768.549470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3768.552612] Lustre: Skipped 1 previous similar message [ 3768.602272] Lustre: lustre-MDT0001: new disk, initializing [ 3768.685725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3768.690806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3770.891629] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3775.791407] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3778.327305] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3781.645099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3781.784267] Lustre: lustre-OST0000: new disk, initializing [ 3781.786521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3781.790124] Lustre: 249083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3781.829843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.833411] Lustre: Skipped 1 previous similar message [ 3784.489469] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3789.268780] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3790.848441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3790.853682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3790.895220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3795.526666] LustreError: 247916:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3800.508394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3800.599605] Lustre: lustre-OST0001: new disk, initializing [ 3800.601740] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3800.606186] Lustre: 250197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3800.649446] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3803.532975] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3805.616088] LustreError: 247916:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3805.628806] LustreError: 247124:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3808.116313] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3808.775192] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3810.315614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3810.322040] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3810.364739] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3815.720294] LustreError: 247124:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3817.246322] Lustre: lustre-OST0001: Not available for connect from 192.168.206.5@tcp (stopping) [ 3820.512842] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.515711] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.519340] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3820.523162] Lustre: Skipped 2 previous similar messages [ 3822.667384] Lustre: server umount lustre-OST0001 complete [ 3825.635282] LustreError: 249064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.640256] LustreError: 249064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3830.757044] LustreError: 249067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3830.764723] LustreError: 249067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3835.876000] LustreError: 249066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3835.876300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.878302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3835.878307] Lustre: Skipped 2 previous similar messages [ 3835.886681] LustreError: 249066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3835.891601] Lustre: Skipped 1 previous similar message [ 3840.996805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3841.003126] Lustre: Skipped 2 previous similar messages [ 3841.381177] Lustre: server umount lustre-OST0000 complete [ 3845.601100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3845.607556] LustreError: Skipped 1 previous similar message [ 3845.611989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.233334] LustreError: 247916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.240731] LustreError: 247916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3851.378976] Lustre: server umount lustre-MDT0000 complete [ 3853.053180] LustreError: 247108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869006 with bad export cookie 17992477544445110128 [ 3853.056541] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3853.059015] LustreError: 247108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3859.300954] Lustre: server umount lustre-MDT0001 complete [ 3863.879613] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 3865.165475] Key type lgssc unregistered [ 3865.309620] LNet: 252235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3865.314735] LNetError: 252235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3865.324033] LNet: Removed LNI 192.168.206.105@tcp [ 3865.698777] Key type .llcrypt unregistered [ 3865.700206] Key type ._llcrypt unregistered [ 3883.091416] Key type ._llcrypt registered [ 3883.092648] Key type .llcrypt registered [ 3883.135068] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 3883.610286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3883.617706] alg: No test for adler32 (adler32-zlib) [ 3884.486388] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 3884.593412] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 3886.192235] Key type lgssc registered [ 3886.644092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3890.003718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3890.017920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.130325] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3891.142801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3891.209869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.748125] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3895.969738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.036475] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3896.051086] Lustre: 254416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3896.171202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3897.677952] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3900.312570] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3902.314938] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3904.862572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.940828] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3905.018734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.289206] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3910.235833] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3913.244939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3959.265466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3959.266404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.270071] LustreError: Skipped 1 previous similar message [ 3959.276116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3959.279216] Lustre: Skipped 1 previous similar message [ 3961.031748] Lustre: server umount lustre-OST0000 complete [ 3963.047215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3968.716538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3968.880125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.761392] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 3975.343142] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3979.111960] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3979.118173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3979.118739] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3979.128856] Lustre: Skipped 1 previous similar message [ 4009.958441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.959248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.960019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.965746] Lustre: Skipped 2 previous similar messages [ 4009.975729] LustreError: Skipped 1 previous similar message [ 4015.077458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4015.080455] Lustre: Skipped 2 previous similar messages [ 4015.340505] Lustre: server umount lustre-OST0000 complete [ 4019.168949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4019.173331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.181888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.133290] Lustre: server umount lustre-MDT0000 complete [ 4024.889793] LustreError: 254408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869178 with bad export cookie 8840028451280543944 [ 4024.893937] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4024.898035] LustreError: 254408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4025.079836] Lustre: server umount lustre-MDT0001 complete [ 4029.461275] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4030.664967] Key type lgssc unregistered [ 4030.823777] LNet: 259829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4030.828601] LNetError: 259829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4030.840365] LNet: Removed LNI 192.168.206.105@tcp [ 4031.250444] Key type .llcrypt unregistered [ 4031.251263] Key type ._llcrypt unregistered [ 4042.714177] Key type ._llcrypt registered [ 4042.715057] Key type .llcrypt registered [ 4042.765401] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4043.220876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4043.237776] alg: No test for adler32 (adler32-zlib) [ 4044.131225] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 4044.260088] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4045.872358] Key type lgssc registered [ 4046.371628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4049.771453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4049.787402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.990553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4052.833854] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4056.731094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.951345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.991154] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4062.486488] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4064.968980] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4067.935273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4068.112849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.828781] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4074.333283] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4076.330945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4096.993578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.997483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.005100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4098.020131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.020959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4098.022292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4102.071264] Lustre: server umount lustre-OST0000 complete [ 4108.260243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4108.264953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4108.267996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4108.274222] Lustre: Skipped 1 previous similar message [ 4108.280886] Lustre: Skipped 2 previous similar messages [ 4110.079936] Lustre: server umount lustre-MDT0000 complete [ 4111.868707] LustreError: 261198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869265 with bad export cookie 10368139435425336551 [ 4111.872115] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.874247] LustreError: 261198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4112.028623] Lustre: server umount lustre-MDT0001 complete [ 4116.887968] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4118.238244] Key type lgssc unregistered [ 4118.383305] LNet: 265538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4118.389422] LNetError: 265538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4118.405988] LNet: Removed LNI 192.168.206.105@tcp [ 4118.800137] Key type .llcrypt unregistered [ 4118.801807] Key type ._llcrypt unregistered [ 4127.941195] Key type ._llcrypt registered [ 4127.942762] Key type .llcrypt registered [ 4127.998845] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4128.443465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.503505] alg: No test for adler32 (adler32-zlib) [ 4129.483426] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 4129.611729] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4131.224192] Key type lgssc registered [ 4131.795634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4135.558292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4135.570124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.764193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.363224] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4142.027903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.191742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4143.881523] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4146.973524] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4149.350556] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4152.310220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4152.451021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.934133] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4158.287647] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4162.026736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4168.160932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4168.164355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4168.171175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4172.263698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4172.265803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4172.277397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.911379] Lustre: server umount lustre-OST0000 complete [ 4177.379468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.381672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.384542] Lustre: Skipped 1 previous similar message [ 4177.387499] Lustre: Skipped 1 previous similar message [ 4182.497891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4182.502201] Lustre: Skipped 2 previous similar messages [ 4187.618070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.622874] Lustre: Skipped 1 previous similar message [ 4190.176216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4190.255853] Lustre: server umount lustre-MDT0000 complete [ 4191.937357] LustreError: 266714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869345 with bad export cookie 14411262864220161346 [ 4191.940071] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4191.945446] LustreError: 266714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4192.076493] Lustre: server umount lustre-MDT0001 complete [ 4196.543210] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4197.752056] Key type lgssc unregistered [ 4197.903469] LNet: 270406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4197.906223] LNetError: 270406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4197.912401] LNet: Removed LNI 192.168.206.105@tcp [ 4198.289992] Key type .llcrypt unregistered [ 4198.291749] Key type ._llcrypt unregistered [ 4209.400928] Key type ._llcrypt registered [ 4209.402432] Key type .llcrypt registered [ 4209.453460] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4209.774668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4209.790081] alg: No test for adler32 (adler32-zlib) [ 4210.665262] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 4210.781389] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4212.376193] Key type lgssc registered [ 4212.856278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4216.732285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4216.742364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.948083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4219.650677] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4223.194763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4223.349042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4225.130151] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4228.348605] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4230.911598] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4234.173622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.345149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.992504] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4240.460698] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4243.565655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4264.419131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4264.421484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.422356] LustreError: Skipped 1 previous similar message [ 4264.423386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.428603] Lustre: Skipped 1 previous similar message [ 4264.441419] Lustre: Skipped 1 previous similar message [ 4267.701155] Lustre: server umount lustre-OST0000 complete [ 4269.536865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4269.541481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.543126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.548432] Lustre: Skipped 1 previous similar message [ 4274.657670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4274.662521] Lustre: Skipped 2 previous similar messages [ 4275.709880] Lustre: server umount lustre-MDT0000 complete [ 4277.751142] LustreError: 271799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869431 with bad export cookie 13253136966093751157 [ 4277.752532] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4277.756174] LustreError: 271799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4277.929079] Lustre: server umount lustre-MDT0001 complete [ 4283.194360] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4284.671329] Key type lgssc unregistered [ 4284.860581] LNet: 276138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4284.865472] LNetError: 276138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4284.875461] LNet: Removed LNI 192.168.206.105@tcp [ 4285.347248] Key type .llcrypt unregistered [ 4285.349574] Key type ._llcrypt unregistered [ 4293.565712] Key type ._llcrypt registered [ 4293.567073] Key type .llcrypt registered [ 4293.615472] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4294.034617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4294.067718] alg: No test for adler32 (adler32-zlib) [ 4294.967064] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 4295.091255] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4296.712149] Key type lgssc registered [ 4297.201764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4300.750589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4300.760985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.004297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.995903] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4308.061756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.285245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4310.276386] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4313.546497] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4316.052589] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4318.792922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4318.958909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.416245] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4324.448212] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4327.143142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4338.658414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.660258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4338.660637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.663112] Lustre: Skipped 1 previous similar message [ 4343.777420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4343.781523] Lustre: Skipped 2 previous similar messages [ 4344.812721] Lustre: server umount lustre-OST0000 complete [ 4348.901489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.902881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.906377] Lustre: Skipped 1 previous similar message [ 4348.911166] Lustre: Skipped 1 previous similar message [ 4352.760884] Lustre: server umount lustre-MDT0000 complete [ 4354.017487] LustreError: 277348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.024485] LustreError: 277348:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4354.490611] LustreError: 277313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869507 with bad export cookie 17396401798950600797 [ 4354.493633] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.495697] LustreError: 277313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4354.677208] Lustre: server umount lustre-MDT0001 complete [ 4359.837278] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4361.199135] Key type lgssc unregistered [ 4361.370540] LNet: 281065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4361.374932] LNetError: 281065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4361.382729] LNet: Removed LNI 192.168.206.105@tcp [ 4361.837479] Key type .llcrypt unregistered [ 4361.839804] Key type ._llcrypt unregistered [ 4366.881374] Key type ._llcrypt registered [ 4366.882682] Key type .llcrypt registered [ 4366.928223] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4374.436603] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 4381.521370] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4382.033454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4382.047346] alg: No test for adler32 (adler32-zlib) [ 4382.956274] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 4383.063621] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4384.666292] Key type lgssc registered [ 4385.189445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4402.204294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4402.214653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.359807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4403.387793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4403.433936] Lustre: lustre-MDT0000: new disk, initializing [ 4403.479325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.489416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4405.140963] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4410.272754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.337609] Lustre: 284583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4410.371756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4410.374351] Lustre: Skipped 1 previous similar message [ 4410.416059] Lustre: lustre-MDT0001: new disk, initializing [ 4410.478038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4410.500033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4410.505083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4412.395978] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4417.257865] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4419.789505] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4422.999300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4423.165380] Lustre: lustre-OST0000: new disk, initializing [ 4423.167685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4423.170348] Lustre: 286552:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4423.209272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.143866] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4427.221783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4427.238059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4427.288301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4431.236413] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4433.492299] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4433.636480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4436.488713] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4436.649561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4442.600065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.600483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4442.602477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4442.608531] Lustre: Skipped 1 previous similar message [ 4442.621908] LustreError: Skipped 1 previous similar message [ 4447.719616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4447.724798] Lustre: Skipped 2 previous similar messages [ 4452.320113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4452.398545] Lustre: server umount lustre-OST0000 complete [ 4456.424222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4456.430291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.446037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.955241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.374972] Lustre: server umount lustre-MDT0000 complete [ 4462.245695] LustreError: 284578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869615 with bad export cookie 6766940065137617845 [ 4462.249731] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4462.257000] LustreError: 284578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4462.427501] Lustre: server umount lustre-MDT0001 complete [ 4466.632648] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:13:39 (1788869619) [ 4470.049045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.331428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.443937] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4475.362069] LustreError: 288960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.373648] LustreError: 288960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4476.441610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.606113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.464616] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4482.301548] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4484.973949] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4488.087817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.271364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4491.458237] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4495.220856] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.082440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4506.087528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.093612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4506.095984] Lustre: Skipped 2 previous similar messages [ 4511.023767] Lustre: server umount lustre-OST0000 complete [ 4512.736995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4512.741177] LustreError: Skipped 1 previous similar message [ 4512.744150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.749517] Lustre: Skipped 1 previous similar message [ 4512.752603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.755431] Lustre: Skipped 2 previous similar messages [ 4518.883700] Lustre: server umount lustre-MDT0000 complete [ 4520.649403] LustreError: 288942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869673 with bad export cookie 6766940065137618734 [ 4520.654391] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.657224] LustreError: 288942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4520.849465] Lustre: server umount lustre-MDT0001 complete [ 4525.364649] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4526.592703] Key type lgssc unregistered [ 4526.746602] LNet: 292491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4526.750280] LNetError: 292491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4526.759444] LNet: Removed LNI 192.168.206.105@tcp [ 4527.107950] Key type .llcrypt unregistered [ 4527.109256] Key type ._llcrypt unregistered [ 4538.578571] Key type ._llcrypt registered [ 4538.579881] Key type .llcrypt registered [ 4538.625216] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 4546.730698] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4547.341420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4547.350062] alg: No test for adler32 (adler32-zlib) [ 4548.259286] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 4548.383816] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4550.000190] Key type lgssc registered [ 4550.543916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4570.063504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4570.076124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.183716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4571.200603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4571.245379] Lustre: lustre-MDT0000: new disk, initializing [ 4571.283421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.295187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4573.154430] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4578.384926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.450745] Lustre: 296285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4578.480728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4578.484030] Lustre: Skipped 1 previous similar message [ 4578.526231] Lustre: lustre-MDT0001: new disk, initializing [ 4578.585790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4578.612049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4578.616672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4580.319980] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4584.708940] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4587.090528] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4589.979478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4590.119428] Lustre: lustre-OST2710: new disk, initializing [ 4590.121599] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4590.125108] Lustre: 298251:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4590.168101] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4592.714064] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4597.180255] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4598.285569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4598.290712] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4598.358909] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4600.295816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4600.377333] Lustre: lustre-OST03e8: new disk, initializing [ 4600.379447] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4600.383585] Lustre: 299314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4600.433585] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4603.088474] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4606.433083] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4607.508131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4607.516169] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4607.557347] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4609.768492] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4609.857484] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4611.622480] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4611.738388] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4613.606222] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4613.699398] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4615.444740] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4615.529190] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4617.700416] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.702264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4617.704536] Lustre: Skipped 3 previous similar messages [ 4617.709393] Lustre: Skipped 3 previous similar messages [ 4622.819724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4622.822156] Lustre: Skipped 2 previous similar messages [ 4623.078244] Lustre: server umount lustre-MDT0000 complete [ 4624.436191] LustreError: 296277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869777 with bad export cookie 17763038554985423646 [ 4624.438734] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.440802] LustreError: 296277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4624.566195] Lustre: server umount lustre-MDT0001 complete [ 4636.174764] Lustre: server umount lustre-OST2710 complete [ 4648.249588] Lustre: server umount lustre-OST03e8 complete [ 4656.505927] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 4660.532545] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4674.556200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4674.655176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4674.670169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4674.707459] Lustre: lustre-MDT0000: new disk, initializing [ 4674.740799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.749187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4676.493114] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4681.824787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.893485] Lustre: 304557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4682.021126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4683.658541] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4688.259720] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4690.836251] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4693.990602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.139276] Lustre: lustre-OST0000: new disk, initializing [ 4694.141437] Lustre: Skipped 1 previous similar message [ 4694.144353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4694.147367] Lustre: Skipped 2 previous similar messages [ 4694.150588] Lustre: 306524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4694.189500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4694.193721] Lustre: Skipped 1 previous similar message [ 4696.756506] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4696.841512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4696.844327] Lustre: Skipped 1 previous similar message [ 4696.846663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4696.906772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4701.279268] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4703.020555] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4703.114136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4704.956888] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4705.043506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4707.299597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.299819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4707.307783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4707.312848] Lustre: Skipped 1 previous similar message [ 4712.030548] Lustre: server umount lustre-OST0000 complete [ 4717.538131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4717.540179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4717.546202] Lustre: Skipped 2 previous similar messages [ 4722.658867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.662059] Lustre: Skipped 2 previous similar messages [ 4728.288140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4728.377931] Lustre: server umount lustre-MDT0000 complete [ 4729.981348] LustreError: 305362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869883 with bad export cookie 17763038554985425312 [ 4729.984749] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4729.987597] LustreError: 305362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4730.134537] Lustre: server umount lustre-MDT0001 complete [ 4733.756533] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:18:06 (1788869886) [ 4734.449991] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4735.209937] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:18:07 (1788869887) [ 4736.725677] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4740.475004] hrtimer: interrupt took 3307931 ns [ 4749.624254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.744199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4749.758314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4749.842043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.791770] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4755.615971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.668812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4755.682724] Lustre: 310377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4757.631976] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4774.825190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4774.935960] Lustre: Denying initial registration attempt from nid 192.168.206.105@tcp, specified as failover [ 4774.940664] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4774.946177] LustreError: 311828:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 4774.951801] LustreError: 311828:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 4774.956115] LustreError: 311828:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4774.988040] Lustre: server umount lustre-OST0000 complete [ 4774.990687] LustreError: 311828:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4777.959555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.962255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.964385] Lustre: Skipped 1 previous similar message [ 4777.968194] Lustre: Skipped 3 previous similar messages [ 4792.288120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4792.357244] Lustre: server umount lustre-MDT0000 complete [ 4793.881962] LustreError: 310370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869947 with bad export cookie 17763038554985426194 [ 4793.884936] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4793.889026] LustreError: 310370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4794.048076] Lustre: server umount lustre-MDT0001 complete [ 4798.521370] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4799.805765] Key type lgssc unregistered [ 4799.970608] LNet: 312948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4799.974361] LNetError: 312948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4799.987509] LNet: Removed LNI 192.168.206.105@tcp [ 4800.437487] Key type .llcrypt unregistered [ 4800.439202] Key type ._llcrypt unregistered [ 4805.106119] Key type ._llcrypt registered [ 4805.108216] Key type .llcrypt registered [ 4805.157916] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4805.542326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4805.575501] alg: No test for adler32 (adler32-zlib) [ 4806.478826] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 4806.593852] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4808.192391] Key type lgssc registered [ 4808.688775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4835.472737] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4839.356099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4839.367152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.493597] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4840.503408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4840.574535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.245903] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4845.706091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.751491] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4845.764275] Lustre: 315681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4845.892199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4847.636144] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4850.559150] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4853.076416] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4868.772744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4868.865944] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4868.944442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4871.076113] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4873.837685] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4874.656478] Lustre: server umount lustre-OST0000 complete [ 4876.769217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4876.772287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.777607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4878.835104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4878.835519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.155432] Lustre: server umount lustre-MDT0000 complete [ 4883.749899] LustreError: 315673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870037 with bad export cookie 13381284500250262617 [ 4883.751275] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4883.755383] LustreError: 315673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4883.915220] Lustre: server umount lustre-MDT0001 complete [ 4888.155151] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 4889.347780] Key type lgssc unregistered [ 4889.507511] LNet: 319246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4889.512235] LNetError: 319246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4889.521598] LNet: Removed LNI 192.168.206.105@tcp [ 4889.879053] Key type .llcrypt unregistered [ 4889.880322] Key type ._llcrypt unregistered [ 4900.478246] Key type ._llcrypt registered [ 4900.479692] Key type .llcrypt registered [ 4900.521657] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 4900.890578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4900.916833] alg: No test for adler32 (adler32-zlib) [ 4901.787410] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 4901.882243] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 4903.472422] Key type lgssc registered [ 4903.957717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4907.436759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4907.446645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.638253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4910.352238] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4913.707343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.901414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4915.613518] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4918.395319] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4920.471781] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4925.028089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4925.187620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.455850] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4930.208367] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4947.734611] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.5@tcp (stopping) [ 4949.985906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4949.987637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4949.994778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4949.996943] Lustre: Skipped 2 previous similar messages [ 4949.998567] Lustre: Skipped 1 previous similar message [ 4951.528496] Lustre: server umount lustre-MDT0000 complete [ 4952.842109] LustreError: 320630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.851332] LustreError: 320630:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4953.247427] LustreError: 320615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870106 with bad export cookie 15710396786168231506 [ 4953.247950] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4953.254962] LustreError: 320615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4955.105884] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.107551] LustreError: 321873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.111380] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4957.962100] LustreError: 320629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4957.973808] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.5@tcp (stopping) [ 4959.450448] Lustre: server umount lustre-MDT0001 complete [ 4962.387828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4966.144032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.656826] LustreError: 319945:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b04f3a89 x1875766206818560/t0(0) o250->MGC192.168.206.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4978.783773] LustreError: 322464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4978.792016] LustreError: 322464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4978.812673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.837273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4980.404757] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4982.882184] LustreError: 324245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4982.890308] LustreError: 324245:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4983.634941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4983.770408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4985.241197] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 4987.879928] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4988.009378] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4989.608694] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 4991.562679] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4999.205834] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5003.234826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.235884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5003.238267] Lustre: Skipped 1 previous similar message [ 5003.241308] Lustre: Skipped 2 previous similar messages [ 5006.625340] Lustre: server umount lustre-OST0000 complete [ 5008.354627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.359194] Lustre: Skipped 1 previous similar message [ 5013.473306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5013.477461] Lustre: Skipped 3 previous similar messages [ 5014.111990] Lustre: server umount lustre-MDT0000 complete [ 5015.586969] LustreError: 320615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870168 with bad export cookie 15710396786168238660 [ 5015.590699] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5015.595614] LustreError: 320615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5015.737115] Lustre: server umount lustre-MDT0001 complete [ 5019.858067] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5021.083671] Key type lgssc unregistered [ 5021.241499] LNet: 326913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5021.245221] LNetError: 326913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5021.257399] LNet: Removed LNI 192.168.206.105@tcp [ 5021.615762] Key type .llcrypt unregistered [ 5021.617200] Key type ._llcrypt unregistered [ 5031.222487] Key type ._llcrypt registered [ 5031.223947] Key type .llcrypt registered [ 5031.263085] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5031.676611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5031.683916] alg: No test for adler32 (adler32-zlib) [ 5032.546547] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 5032.633793] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 5034.216179] Key type lgssc registered [ 5034.598658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5037.466890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5037.474295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.677830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.345719] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5043.637094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.799189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5045.392789] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5048.187076] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5050.365168] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5053.409567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5053.410113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5053.413682] Lustre: Skipped 1 previous similar message [ 5054.433195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5054.435018] Lustre: Skipped 2 previous similar messages [ 5057.628708] Lustre: server umount lustre-MDT0000 complete [ 5058.810254] LustreError: 328283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870212 with bad export cookie 6025677812401518374 [ 5058.814037] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5058.814095] LustreError: 328283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5058.917488] Lustre: server umount lustre-MDT0001 complete [ 5063.680265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.774096] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5063.786628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5063.855096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.256427] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5068.431074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.468879] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5068.481578] Lustre: 330887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5068.588990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5070.028483] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5072.846607] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5074.803274] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5079.653045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5079.719782] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5079.724406] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5079.727185] LustreError: 332768:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5079.729734] LustreError: 332768:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 5079.731929] LustreError: 332768:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5079.757547] Lustre: server umount lustre-OST0000 complete [ 5079.758857] LustreError: 332768:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5084.963195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5085.041915] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5085.120237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.190280] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5089.963384] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5091.867185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5092.887707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5092.922286] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5092.965931] Lustre: lustre-OST0001: new disk, initializing [ 5092.968813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5092.973127] Lustre: 334309:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5095.318600] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5098.203435] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5099.326304] Lustre: server umount lustre-OST0001 complete [ 5101.042383] LustreError: 333232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.043036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.045161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5101.048770] LustreError: 333232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5101.053013] Lustre: Skipped 1 previous similar message [ 5106.146540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5106.147675] LustreError: 333232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.148866] Lustre: Skipped 2 previous similar messages [ 5106.154339] LustreError: 333232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5111.267476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5111.268474] LustreError: 335309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.270019] Lustre: Skipped 1 previous similar message [ 5111.275746] LustreError: 335309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5115.360205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5115.419690] Lustre: server umount lustre-OST0000 complete [ 5119.968951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5119.972476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.977910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.232109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5131.299946] Lustre: server umount lustre-MDT0000 complete [ 5131.745043] LustreError: 332146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.751302] LustreError: 332146:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5132.568204] LustreError: 330877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870285 with bad export cookie 6025677812401518913 [ 5132.570506] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.579564] LustreError: 330877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5132.690077] Lustre: server umount lustre-MDT0001 complete [ 5136.225086] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5137.236155] Key type lgssc unregistered [ 5137.361434] LNet: 336262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5137.364925] LNetError: 336262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5137.376993] LNet: Removed LNI 192.168.206.105@tcp [ 5137.690348] Key type .llcrypt unregistered [ 5137.691670] Key type ._llcrypt unregistered [ 5169.237110] Key type ._llcrypt registered [ 5169.239038] Key type .llcrypt registered [ 5169.281650] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 5176.430104] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5176.736464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5176.747451] alg: No test for adler32 (adler32-zlib) [ 5177.622040] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 5177.720389] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 5179.312202] Key type lgssc registered [ 5179.735590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5192.525983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5192.536530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.628882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5193.640147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5193.679936] Lustre: lustre-MDT0000: new disk, initializing [ 5193.712584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.720533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5195.314760] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5200.142825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.189538] Lustre: 341158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5200.222927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5200.225806] Lustre: Skipped 1 previous similar message [ 5200.269516] Lustre: lustre-MDT0001: new disk, initializing [ 5200.314937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5200.332546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5200.338669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5201.991377] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5206.125466] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5208.278758] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5210.880403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5210.992086] Lustre: lustre-OST0000: new disk, initializing [ 5210.994200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5210.998641] Lustre: 343130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5211.036338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5213.302174] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5216.768192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5216.774920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5216.809275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5217.408265] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5218.995065] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5219.077994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5220.606797] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5220.691491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5221.858804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5221.859564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.862179] LustreError: Skipped 1 previous similar message [ 5221.868888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5221.872849] Lustre: Skipped 1 previous similar message [ 5223.450316] Lustre: server umount lustre-OST0000 complete [ 5225.952927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5225.956602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.962381] Lustre: Skipped 1 previous similar message [ 5225.964709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5226.977163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5226.977854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.097711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5232.103348] Lustre: Skipped 2 previous similar messages [ 5237.216997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5237.219014] Lustre: Skipped 1 previous similar message [ 5239.264097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5239.324589] Lustre: server umount lustre-MDT0000 complete [ 5240.529112] LustreError: 341954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870393 with bad export cookie 11115179617688977751 [ 5240.532822] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5240.533281] LustreError: 341954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5240.648924] Lustre: server umount lustre-MDT0001 complete [ 5243.462416] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:26:36 (1788870396) [ 5245.961771] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:26:38 (1788870398) [ 5248.777086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5248.950876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5250.328811] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5253.246092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.375112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5254.858704] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5257.357128] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5259.389921] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5261.848046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5261.979604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5264.045570] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5266.653327] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5280.500562] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5281.166779] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5281.792687] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5282.475446] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5283.220352] Lustre: Failing over lustre-MDT0000 [ 5283.432838] Lustre: server umount lustre-MDT0000 complete [ 5284.321219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5284.324651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.330679] LustreError: 345880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.338564] LustreError: 345880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5285.857877] LustreError: 345881:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.863567] LustreError: 345881:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5287.188557] LustreError: 345877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.977565] LustreError: 346607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.984493] LustreError: 346607:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5296.097355] LustreError: 345880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.104297] LustreError: 345880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5296.997943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.072449] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5297.206440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5297.241548] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5297.419606] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5298.930601] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5302.247869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5302.262832] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5302.280424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5304.482208] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5305.212546] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5307.335619] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5308.174178] Lustre: Failing over lustre-MDT0000 [ 5308.359095] Lustre: server umount lustre-MDT0000 complete [ 5312.481810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5312.482492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5312.483032] LustreError: 347878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5312.497065] Lustre: Skipped 3 previous similar messages [ 5321.822700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.896122] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5322.048529] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5323.019977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5323.740479] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5327.334583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5327.336962] Lustre: Skipped 2 previous similar messages [ 5327.346227] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5327.365076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5329.596695] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5330.251422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5337.570936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.572675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5337.572964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5337.575217] Lustre: Skipped 2 previous similar messages [ 5337.577730] Lustre: Skipped 1 previous similar message [ 5338.984211] Lustre: server umount lustre-MDT0000 complete [ 5340.671582] LustreError: 345861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870493 with bad export cookie 11115179617688980432 [ 5340.675613] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5340.676558] LustreError: 345861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5340.821285] Lustre: server umount lustre-MDT0001 complete [ 5352.680442] Lustre: server umount lustre-OST0000 complete [ 5358.978080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5359.216056] LustreError: 352232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.228295] LustreError: 352232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 5359.254288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5359.257965] Lustre: Skipped 1 previous similar message [ 5361.113375] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5364.684993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5366.532145] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5369.675036] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5371.862929] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5374.854829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5377.925366] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5381.289318] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5384.230491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5389.283435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5389.284143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.286742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5389.286748] Lustre: Skipped 1 previous similar message [ 5389.295382] Lustre: Skipped 1 previous similar message [ 5390.387854] Lustre: server umount lustre-OST0000 complete [ 5394.149058] Lustre: server umount lustre-MDT0000 complete [ 5394.405325] LustreError: 353982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.414380] LustreError: 353982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5395.822381] LustreError: 352212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870549 with bad export cookie 11115179617688981181 [ 5395.824193] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.827681] LustreError: 352212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5395.966561] Lustre: server umount lustre-MDT0001 complete [ 5400.301409] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5401.476703] Key type lgssc unregistered [ 5401.611121] LNet: 355759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5401.613704] LNetError: 355759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5401.625334] LNet: Removed LNI 192.168.206.105@tcp [ 5401.967878] Key type .llcrypt unregistered [ 5401.969219] Key type ._llcrypt unregistered [ 5413.527814] Key type ._llcrypt registered [ 5413.529363] Key type .llcrypt registered [ 5413.584087] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5413.965297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5414.021985] alg: No test for adler32 (adler32-zlib) [ 5414.939673] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 5415.053272] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 5416.656215] Key type lgssc registered [ 5417.180030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5420.699586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5420.714078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.924110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.641552] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5427.220051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.374698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5429.064476] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5432.402943] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5434.643063] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5437.112135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5437.234052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5439.359268] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5442.276106] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5447.465178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5452.769920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.770852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5452.773850] Lustre: Skipped 2 previous similar messages [ 5452.775660] Lustre: Skipped 1 previous similar message [ 5457.823173] Lustre: server umount lustre-MDT0000 complete [ 5457.889953] LustreError: 357244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.895954] LustreError: 357244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5459.201092] LustreError: 359825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870612 with bad export cookie 17777525276715858822 [ 5459.204233] LustreError: 359825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5459.204335] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.329755] Lustre: server umount lustre-MDT0001 complete [ 5471.145161] Lustre: server umount lustre-OST0000 complete [ 5476.284187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.443068] LustreError: 360953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.466066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.852399] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5480.732567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.866037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5482.335663] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5484.898353] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5486.842625] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5489.245537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5489.381835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5491.583668] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5494.557446] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5501.673857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5516.769148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5516.772410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5516.777957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5516.780761] Lustre: Skipped 1 previous similar message [ 5518.499600] Lustre: server umount lustre-MDT0000 complete [ 5519.951116] LustreError: 360935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870673 with bad export cookie 17777525276715859921 [ 5519.954198] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5519.955071] LustreError: 360935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5520.077117] Lustre: server umount lustre-MDT0001 complete [ 5531.745876] Lustre: server umount lustre-OST0000 complete [ 5537.655191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.824306] LustreError: 364882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.852233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5539.454420] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5542.880901] LustreError: 364883:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.949453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.692751] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5547.643494] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5549.917736] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5552.471260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5554.741581] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5557.486914] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5562.855496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5567.971314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5567.971645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.978886] Lustre: Skipped 2 previous similar messages [ 5567.981686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5567.984305] Lustre: Skipped 4 previous similar messages [ 5570.587983] Lustre: server umount lustre-OST0000 complete [ 5573.089655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.090659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5573.094179] Lustre: Skipped 2 previous similar messages [ 5578.209060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.211582] Lustre: Skipped 2 previous similar messages [ 5586.400343] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5586.462985] Lustre: server umount lustre-MDT0000 complete [ 5587.885431] LustreError: 364864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870741 with bad export cookie 17777525276715860929 [ 5587.889708] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5587.891828] LustreError: 364864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5588.020678] Lustre: server umount lustre-MDT0001 complete [ 5592.138944] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5593.398863] Key type lgssc unregistered [ 5593.551450] LNet: 368412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5593.553995] LNetError: 368412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5593.562530] LNet: Removed LNI 192.168.206.105@tcp [ 5593.954788] Key type .llcrypt unregistered [ 5593.956507] Key type ._llcrypt unregistered [ 5605.123800] Key type ._llcrypt registered [ 5605.125309] Key type .llcrypt registered [ 5605.166317] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5605.514752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5605.590354] alg: No test for adler32 (adler32-zlib) [ 5606.461521] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 5606.564641] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 5608.160452] Key type lgssc registered [ 5608.599815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5611.576908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5611.583259] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.586498] LustreError: 369826:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5611.595271] LustreError: 369826:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5611.597419] LustreError: 369826:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5611.601645] LustreError: 369826:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5611.604668] LustreError: 369826:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5614.435745] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5614.441444] LustreError: 370222:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5614.455187] LustreError: 370222:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5614.458357] LustreError: 370222:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5614.462074] LustreError: 370222:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5614.466370] LustreError: 370222:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5622.677130] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5623.883823] Key type lgssc unregistered [ 5624.025476] LNet: 371224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5624.028807] LNetError: 371224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5624.038425] LNet: Removed LNI 192.168.206.105@tcp [ 5624.382364] Key type .llcrypt unregistered [ 5624.383967] Key type ._llcrypt unregistered [ 5631.797432] Key type ._llcrypt registered [ 5631.798689] Key type .llcrypt registered [ 5631.840738] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 5638.416017] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5638.920290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5638.929469] alg: No test for adler32 (adler32-zlib) [ 5639.815195] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 5639.923568] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 5641.520339] Key type lgssc registered [ 5641.978557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5655.395734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5655.407295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5656.546897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5656.566208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5656.613958] Lustre: lustre-MDT0000: new disk, initializing [ 5656.655568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5656.673246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5658.352681] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5663.289171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.342378] Lustre: 374348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5663.369243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5663.371762] Lustre: Skipped 1 previous similar message [ 5663.411721] Lustre: lustre-MDT0001: new disk, initializing [ 5663.460223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5663.479022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5663.485137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5665.218712] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5669.594713] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5671.964967] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5674.914130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5675.058702] Lustre: lustre-OST0000: new disk, initializing [ 5675.061453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5675.064950] Lustre: 376316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5675.100794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5677.528403] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5680.641260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5680.645380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5680.689164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5681.518573] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5683.020885] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5683.096464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5684.487706] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5684.559382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5685.729529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5685.731532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5685.734814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5685.739965] Lustre: Skipped 1 previous similar message [ 5687.317993] Lustre: server umount lustre-OST0000 complete [ 5689.312954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5689.316579] LustreError: Skipped 1 previous similar message [ 5689.318347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5689.322732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5689.325341] Lustre: Skipped 1 previous similar message [ 5690.849815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5690.850114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.750536] Lustre: server umount lustre-MDT0000 complete [ 5695.985350] LustreError: 374339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870849 with bad export cookie 3482922341509101275 [ 5695.989400] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5695.993112] LustreError: 374339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5696.111680] Lustre: server umount lustre-MDT0001 complete [ 5699.168489] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5699.815282] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:34:12 (1788870852) [ 5702.236206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.390617] LustreError: 378775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.397934] LustreError: 378775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5702.416220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.837043] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5706.826508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.968352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5708.458654] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5711.116895] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5713.292990] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5715.915813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5716.069793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.266825] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5720.931398] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5723.575546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5723.656367] Lustre: lustre-OST0001: new disk, initializing [ 5723.659200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5723.663798] Lustre: 381605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5726.151927] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5726.163197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5726.168996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5726.219802] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5730.286561] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5732.323809] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.325097] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5732.327727] Lustre: Skipped 1 previous similar message [ 5732.329770] Lustre: Skipped 1 previous similar message [ 5736.725288] Lustre: lustre-OST0001: Not available for connect from 192.168.206.5@tcp (stopping) [ 5736.727274] Lustre: Skipped 1 previous similar message [ 5738.302443] Lustre: server umount lustre-OST0001 complete [ 5742.564403] LustreError: 380593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.570599] LustreError: 380593:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5747.685688] LustreError: 381970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.691475] LustreError: 381970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5752.803257] LustreError: 382072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5752.803437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.803780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5752.804364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5752.804368] Lustre: Skipped 2 previous similar messages [ 5752.808995] LustreError: 382072:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5752.814266] Lustre: Skipped 1 previous similar message [ 5756.500676] Lustre: server umount lustre-OST0000 complete [ 5772.256105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5772.327426] Lustre: server umount lustre-MDT0000 complete [ 5773.281535] LustreError: 378770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.288463] LustreError: 378770:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5773.661265] LustreError: 378756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870926 with bad export cookie 3482922341509102143 [ 5773.664061] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5773.665849] LustreError: 378756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5779.835200] Lustre: server umount lustre-MDT0001 complete [ 5783.710297] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5784.804405] Key type lgssc unregistered [ 5784.937563] LNet: 383589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5784.940759] LNetError: 383589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5784.948363] LNet: Removed LNI 192.168.206.105@tcp [ 5785.286501] Key type .llcrypt unregistered [ 5785.287671] Key type ._llcrypt unregistered [ 5805.623211] Key type ._llcrypt registered [ 5805.624772] Key type .llcrypt registered [ 5805.668588] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 5805.999312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5806.007473] alg: No test for adler32 (adler32-zlib) [ 5806.874420] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 5806.982837] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 5808.576208] Key type lgssc registered [ 5809.019300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5812.203310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5812.213240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.335264] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5813.344853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5813.424514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.029576] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5818.438075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.492678] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5818.506228] Lustre: 386356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5818.621516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.185928] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5822.880846] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5824.897603] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5829.089044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5829.091715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.097308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5830.626685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.626885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5830.638103] Lustre: Skipped 1 previous similar message [ 5831.786561] Lustre: server umount lustre-MDT0000 complete [ 5833.425352] LustreError: 387100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870986 with bad export cookie 17973934016275802585 [ 5833.428461] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5833.431035] LustreError: 387100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5833.550773] Lustre: server umount lustre-MDT0001 complete [ 5837.593760] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:36:30 (1788870990) [ 5840.413445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.638088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.331395] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5845.867799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5846.019631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.585944] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5850.415793] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5852.490903] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5855.021524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5855.104917] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5855.183672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.566148] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5860.575245] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5870.112776] Lustre: Permanently deactivating lustre-OST0000 [ 5870.118811] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5870.836512] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5872.280209] LustreError: 391706:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5872.284442] LustreError: 391706:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5872.288612] LustreError: 391706:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5873.228643] LustreError: 391755:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5873.231571] LustreError: 391755:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5880.153170] Lustre: server umount lustre-OST0000 complete [ 5881.281875] LustreError: 392004:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5881.286618] LustreError: 392004:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5882.850174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.851309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5882.856784] Lustre: Skipped 1 previous similar message [ 5882.861891] Lustre: Skipped 1 previous similar message [ 5887.970333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5887.973020] Lustre: Skipped 1 previous similar message [ 5892.095128] Lustre: server umount lustre-MDT0000 complete [ 5893.090339] LustreError: 388863:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.096445] LustreError: 388863:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5893.451306] LustreError: 388850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871046 with bad export cookie 17973934016275803159 [ 5893.454078] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5893.457121] LustreError: 388850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5902.873391] Lustre: server umount lustre-MDT0001 complete [ 5906.154031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.698911] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5908.778161] LustreError: 393162:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5911.272205] Lustre: 393501:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.105@tcp [ 5912.051330] Lustre: server umount MGS complete [ 5916.704505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.905190] LustreError: 394225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.912325] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5916.915071] Lustre: Skipped 1 previous similar message [ 5916.945108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.571740] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5921.875561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.021512] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5923.757734] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5926.828810] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5929.162694] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5929.916733] Lustre: Permanently reactivating lustre-OST0000 [ 5929.924077] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5929.927088] Lustre: Skipped 2 previous similar messages [ 5953.504144] LustreError: 395956:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5956.598215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5956.738825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.741617] Lustre: Skipped 1 previous similar message [ 5959.102093] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 5962.280631] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5978.596919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.597216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5978.602475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5978.605582] LustreError: Skipped 1 previous similar message [ 5980.373322] Lustre: server umount lustre-OST0000 complete [ 5983.715176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5983.715403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.723651] Lustre: Skipped 2 previous similar messages [ 5988.012823] Lustre: server umount lustre-MDT0000 complete [ 5988.834302] LustreError: 394226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.840682] LustreError: 394226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5989.664320] LustreError: 394205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871142 with bad export cookie 17973934016275804195 [ 5989.667948] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5989.670137] LustreError: 394205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5989.808851] Lustre: server umount lustre-MDT0001 complete [ 5994.254099] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 5995.405194] Key type lgssc unregistered [ 5995.551507] LNet: 398797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5995.554972] LNetError: 398797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5995.563575] LNet: Removed LNI 192.168.206.105@tcp [ 5995.903644] Key type .llcrypt unregistered [ 5995.905371] Key type ._llcrypt unregistered [ 6002.879565] Key type ._llcrypt registered [ 6002.881233] Key type .llcrypt registered [ 6002.925128] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 6010.083569] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6010.431527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6010.438923] alg: No test for adler32 (adler32-zlib) [ 6011.306188] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 6011.414273] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 6013.008175] Key type lgssc registered [ 6013.463379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6026.351635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6026.362177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.457941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6027.473900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6027.520889] Lustre: lustre-MDT0000: new disk, initializing [ 6027.582339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.589673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6029.224777] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6034.433521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.495099] Lustre: 401923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6034.534619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6034.536512] Lustre: Skipped 1 previous similar message [ 6034.579186] Lustre: lustre-MDT0001: new disk, initializing [ 6034.630613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6034.650581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6034.657877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6036.580808] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6041.522125] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6044.134543] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6047.105624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6047.235633] Lustre: lustre-OST0000: new disk, initializing [ 6047.238846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6047.243488] Lustre: 403893:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6047.287268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.813889] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6051.134835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6051.139222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6051.178385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6053.957382] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6055.524887] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6055.610486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6057.083588] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6057.161727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6061.539539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6061.540347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.542015] LustreError: Skipped 1 previous similar message [ 6061.546479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6061.548805] Lustre: Skipped 1 previous similar message [ 6066.658785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6066.661533] Lustre: Skipped 1 previous similar message [ 6071.779669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6071.781698] Lustre: Skipped 1 previous similar message [ 6072.288120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6072.355444] Lustre: server umount lustre-OST0000 complete [ 6075.873079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6075.875897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.880437] Lustre: Skipped 1 previous similar message [ 6075.882746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6076.897803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.017772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6082.020599] Lustre: Skipped 3 previous similar messages [ 6088.160174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6088.228565] Lustre: server umount lustre-MDT0000 complete [ 6089.602446] LustreError: 401915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871242 with bad export cookie 3851158736138284542 [ 6089.608327] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6089.608387] LustreError: 401915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6089.753798] Lustre: server umount lustre-MDT0001 complete [ 6092.822123] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6093.524297] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:40:46 (1788871246) [ 6096.203490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.415136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6097.858919] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6100.978348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.105354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6102.634286] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6105.248938] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6107.088318] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6109.550895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6109.677478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6111.892821] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6114.658236] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6115.582191] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6121.333991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6126.049299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6126.053489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6126.055170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6126.058954] Lustre: Skipped 1 previous similar message [ 6126.062202] Lustre: Skipped 2 previous similar messages [ 6129.115840] Lustre: server umount lustre-OST0000 complete [ 6131.169465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.174069] Lustre: Skipped 1 previous similar message [ 6136.608217] Lustre: server umount lustre-MDT0000 complete [ 6138.016176] LustreError: 409118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871291 with bad export cookie 3851158736138285424 [ 6138.019816] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6138.020557] LustreError: 409118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6138.146500] Lustre: server umount lustre-MDT0001 complete [ 6142.335372] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6143.506229] Key type lgssc unregistered [ 6143.652442] LNet: 410068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6143.655270] LNetError: 410068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6143.663497] LNet: Removed LNI 192.168.206.105@tcp [ 6144.014263] Key type .llcrypt unregistered [ 6144.015544] Key type ._llcrypt unregistered [ 6152.993609] Key type ._llcrypt registered [ 6152.994906] Key type .llcrypt registered [ 6153.044191] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6153.168482] Key type .llcrypt unregistered [ 6153.170080] Key type ._llcrypt unregistered [ 6161.038124] Key type ._llcrypt registered [ 6161.039735] Key type .llcrypt registered [ 6161.085423] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6161.532077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6161.566466] alg: No test for adler32 (adler32-zlib) [ 6162.468637] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 6162.576229] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 6164.160261] Key type lgssc registered [ 6164.586324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6167.704464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6167.712490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.868970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.356868] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6173.301762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.432078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.617322] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6178.145931] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6178.785546] LustreError: 412248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.790743] LustreError: 412248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6179.810150] LustreError: 412267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.814783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6179.816026] LustreError: 412267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6180.446715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.573083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6181.989773] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6190.049080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6190.053098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.061330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6191.074116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6191.074284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6191.077380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.928230] Lustre: server umount lustre-OST0000 complete [ 6199.780512] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6199.784703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.790246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6199.792222] Lustre: Skipped 1 previous similar message [ 6203.623674] Lustre: server umount lustre-MDT0000 complete [ 6205.131265] LustreError: 412274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871358 with bad export cookie 7281699876283849654 [ 6205.134268] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6205.137049] LustreError: 412274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6205.270376] Lustre: server umount lustre-MDT0001 complete [ 6209.602602] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6210.812530] Key type lgssc unregistered [ 6210.958452] LNet: 415390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6210.961405] LNetError: 415390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6210.974338] LNet: Removed LNI 192.168.206.105@tcp [ 6211.332100] Key type .llcrypt unregistered [ 6211.333563] Key type ._llcrypt unregistered [ 6221.364084] Key type ._llcrypt registered [ 6221.365321] Key type .llcrypt registered [ 6221.408469] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6221.781880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6221.833420] alg: No test for adler32 (adler32-zlib) [ 6222.715393] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 6222.832610] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 6224.424189] Key type lgssc registered [ 6224.818883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6227.756913] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6227.763784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6243.232647] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6248.352363] LustreError: 416754:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.105@tcp: failed processing log, type 4: rc = -110 [ 6274.976151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.863189] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6279.345034] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6281.852853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6282.064194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.093479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6283.624348] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6286.773986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.923489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6288.526956] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6291.348995] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6293.451557] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6297.057170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6297.060063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.067374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6297.569049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6297.573632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.580393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.643145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.645505] Lustre: Skipped 1 previous similar message [ 6302.060748] Lustre: server umount lustre-OST0000 complete [ 6305.762724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6305.763923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.770492] Lustre: Skipped 1 previous similar message [ 6305.774389] Lustre: Skipped 1 previous similar message [ 6309.718557] Lustre: server umount lustre-MDT0000 complete [ 6310.881840] LustreError: 417796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.888902] LustreError: 417796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6311.005164] LustreError: 419281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871464 with bad export cookie 11566946073072435383 [ 6311.006481] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.008268] LustreError: 419281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6311.122384] Lustre: server umount lustre-MDT0001 complete [ 6314.880144] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6315.927290] Key type lgssc unregistered [ 6316.073475] LNet: 420319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6316.076325] LNetError: 420319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6316.082413] LNet: Removed LNI 192.168.206.105@tcp [ 6316.430347] Key type .llcrypt unregistered [ 6316.431697] Key type ._llcrypt unregistered [ 6326.234635] Key type ._llcrypt registered [ 6326.236121] Key type .llcrypt registered [ 6326.283990] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6326.733587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6326.775253] alg: No test for adler32 (adler32-zlib) [ 6327.660451] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 6327.766982] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 6329.352260] Key type lgssc registered [ 6329.844891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6333.103678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6333.111716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.312345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.876131] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6339.248495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.425561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6341.027307] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6343.947179] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6346.086981] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6348.778570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6348.925445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.240331] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6354.095744] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6359.144031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6364.435550] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.5@tcp (stopping) [ 6364.643127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6364.647188] Lustre: Skipped 2 previous similar messages [ 6366.235728] Lustre: server umount lustre-MDT0000 complete [ 6367.169793] LustreError: 421688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.5@tcp arrived at 1788871520 with bad export cookie 3692984589117009618 [ 6367.177191] LustreError: 421688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6367.982794] LustreError: 421690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871521 with bad export cookie 3692984589117008967 [ 6367.990102] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6369.761067] LustreError: 421707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.761621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.766659] LustreError: 421707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6369.771068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.775747] Lustre: Skipped 3 previous similar messages [ 6372.769718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.770081] LustreError: 421708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.779466] LustreError: 421708:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6374.100212] Lustre: server umount lustre-OST0000 complete [ 6376.730734] Lustre: server umount lustre-MDT0001 complete [ 6380.994654] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6382.187416] Key type lgssc unregistered [ 6382.337584] LNet: 425388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6382.341984] LNetError: 425388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6382.350653] LNet: Removed LNI 192.168.206.105@tcp [ 6382.739652] Key type .llcrypt unregistered [ 6382.741392] Key type ._llcrypt unregistered [ 6393.489583] Key type ._llcrypt registered [ 6393.490760] Key type .llcrypt registered [ 6393.535018] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6393.906490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6393.915231] alg: No test for adler32 (adler32-zlib) [ 6394.794728] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 6394.897294] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 6396.488201] Key type lgssc registered [ 6396.938470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6400.537609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6400.550993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6401.791216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6403.939961] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6407.969341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6408.172555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6409.961167] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6413.025508] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6415.358479] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6418.229711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6418.370181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6420.794463] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6424.082326] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6427.629422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6432.739146] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6432.741286] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.742555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6432.749671] Lustre: Skipped 1 previous similar message [ 6435.606185] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.5@tcp (stopping) [ 6435.609412] Lustre: Skipped 1 previous similar message [ 6437.542666] Lustre: server umount lustre-MDT0001 complete [ 6437.861473] LustreError: 428748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.869289] LustreError: 428748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6442.977879] LustreError: 427504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6442.978827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.979214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6442.980071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6445.914915] Lustre: server umount lustre-OST0000 complete [ 6447.491870] Lustre: server umount lustre-MDT0000 complete [ 6452.693694] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6453.908269] Key type lgssc unregistered [ 6454.051517] LNet: 430458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6454.056437] LNetError: 430458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6454.064833] LNet: Removed LNI 192.168.206.105@tcp [ 6454.447543] Key type .llcrypt unregistered [ 6454.448847] Key type ._llcrypt unregistered [ 6463.401831] Key type ._llcrypt registered [ 6463.403052] Key type .llcrypt registered [ 6463.445172] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 6470.391735] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6470.736121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6470.778361] alg: No test for adler32 (adler32-zlib) [ 6471.648524] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 6471.752107] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 6473.345422] Key type lgssc registered [ 6473.835774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6488.775390] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6494.551469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6494.560657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.657416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6495.668439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6495.704148] Lustre: lustre-MDT0000: new disk, initializing [ 6495.727288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6495.734674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6497.243927] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6504.016231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6504.131322] Lustre: lustre-OST0000: new disk, initializing [ 6504.134384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6504.137459] Lustre: Skipped 1 previous similar message [ 6504.139322] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6504.141079] Lustre: 435461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6504.166786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6506.331279] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6509.558319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6509.562512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6509.575061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6510.416747] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6516.562401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.612533] Lustre: 434439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6516.647176] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6516.693420] Lustre: lustre-MDT0001: new disk, initializing [ 6516.746502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6516.761642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6516.766216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6518.394418] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6522.557811] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6522.646533] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6524.353749] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6524.449337] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6528.282086] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.5@tcp (stopping) [ 6528.545181] Lustre: server umount lustre-MDT0001 complete [ 6530.021883] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.022358] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6530.028036] Lustre: Skipped 1 previous similar message [ 6535.140159] LustreError: 434447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.149023] LustreError: 434447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6540.258406] LustreError: 434447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.265933] LustreError: 434447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6545.378232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6545.379390] LustreError: 437154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.381637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6545.382192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6547.549900] Lustre: server umount lustre-OST0000 complete [ 6555.337218] Lustre: server umount lustre-MDT0000 complete [ 6561.056883] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6562.212279] Key type lgssc unregistered [ 6562.378610] LNet: 439199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6562.382150] LNetError: 439199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6562.393435] LNet: Removed LNI 192.168.206.105@tcp [ 6562.755234] Key type .llcrypt unregistered [ 6562.756694] Key type ._llcrypt unregistered [ 6580.011101] Key type ._llcrypt registered [ 6580.012556] Key type .llcrypt registered [ 6580.063044] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 6580.197476] Key type .llcrypt unregistered [ 6580.199616] Key type ._llcrypt unregistered [ 6588.460369] Key type ._llcrypt registered [ 6588.463459] Key type .llcrypt registered [ 6588.507586] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 6588.977165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6589.011594] alg: No test for adler32 (adler32-zlib) [ 6589.916179] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 6590.032874] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 6591.632198] Key type lgssc registered [ 6592.225109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6604.746126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6604.755308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.856540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6605.871891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6605.912468] Lustre: lustre-MDT0000: new disk, initializing [ 6605.948813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.956867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6607.833282] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6612.690805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6612.742935] Lustre: 442542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6612.775555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6612.777988] Lustre: Skipped 1 previous similar message [ 6612.818133] Lustre: lustre-MDT0001: new disk, initializing [ 6612.863633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6612.878613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6612.882949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6614.477059] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6618.841976] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6621.080516] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6623.483663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6623.596878] Lustre: lustre-OST0000: new disk, initializing [ 6623.599095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6623.602232] Lustre: 444511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6623.632114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6623.840905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6623.844456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6623.879118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6625.625612] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6629.510848] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6633.441080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6633.444626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6633.450181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.467712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.468523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.472352] Lustre: Skipped 1 previous similar message [ 6634.480728] Lustre: Skipped 1 previous similar message [ 6638.119688] Lustre: server umount lustre-MDT0000 complete [ 6639.591256] LustreError: 444510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871792 with bad export cookie 5446361454133518518 [ 6639.594098] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6639.596453] LustreError: 444510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6639.742558] Lustre: server umount lustre-MDT0001 complete [ 6651.621201] Lustre: server umount lustre-OST0000 complete [ 6655.708453] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:50:08 (1788871808) [ 6658.722167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.905362] LustreError: 446537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6658.912234] LustreError: 446537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6658.931581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6660.527219] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6663.900036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.082657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6665.827843] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6669.005893] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6671.665040] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6687.716820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6687.800994] Lustre: Found index 0 for lustre-OST0000, updating log [ 6687.886077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6690.592880] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 6694.336020] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6705.328127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6714.849536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6714.852729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.858560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6714.861826] Lustre: Skipped 1 previous similar message [ 6722.018967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.021255] Lustre: Skipped 4 previous similar messages [ 6726.112149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6726.184281] Lustre: server umount lustre-OST0000 complete [ 6740.963623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6740.970183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.981063] Lustre: Skipped 1 previous similar message [ 6740.984681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6745.374723] Lustre: server umount lustre-MDT0000 complete [ 6747.013032] LustreError: 446518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871900 with bad export cookie 5446361454133519897 [ 6747.015762] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6747.019053] LustreError: 446518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6747.162681] Lustre: server umount lustre-MDT0001 complete [ 6750.546131] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:51:43 (1788871903) [ 6751.840256] LNet: 1 local NIs in recovery (showing 1): 192.168.206.105@tcp [ 7335.403274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.601654] LustreError: 452971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.632766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.331887] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7340.830992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.972221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7342.441870] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7345.078887] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7347.174721] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7362.723556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7362.834936] Lustre: Found index 0 for lustre-OST0000, updating log [ 7362.948484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7365.169404] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7368.003195] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7373.282047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7373.282674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7373.294289] Lustre: Skipped 1 previous similar message [ 7373.297111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7373.300405] Lustre: Skipped 2 previous similar messages [ 7374.938037] Lustre: server umount lustre-OST0000 complete [ 7376.864771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7376.867464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.874820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.890506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.913613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7378.917133] Lustre: Skipped 3 previous similar messages [ 7382.617069] Lustre: server umount lustre-MDT0000 complete [ 7383.009903] LustreError: 452990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7383.017123] LustreError: 452990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7383.928076] LustreError: 452952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872537 with bad export cookie 5446361454133520968 [ 7383.931097] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7383.932404] LustreError: 452952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7384.041616] Lustre: server umount lustre-MDT0001 complete [ 7390.816262] LNet: 1 local NIs in recovery (showing 1): 192.168.206.105@tcp [ 7402.113907] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 7405.886217] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7420.395394] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:02:53 (1788872573) [ 7427.040808] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7428.163353] Key type lgssc unregistered [ 7428.317371] LNet: 459610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7428.321129] LNetError: 459610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7492.838319] LNet: Removed LNI 192.168.206.105@tcp [ 7493.181630] Key type .llcrypt unregistered [ 7493.183056] Key type ._llcrypt unregistered [ 7500.439843] Key type ._llcrypt registered [ 7500.441014] Key type .llcrypt registered [ 7500.478835] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7500.812914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7500.882142] alg: No test for adler32 (adler32-zlib) [ 7501.747812] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 7501.841087] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7503.424342] Key type lgssc registered [ 7503.825423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7505.367176] LNet: Added LNI 192.168.206.105@tcp100 [8/256/0/180] [ 7505.692987] LNet: Added LNI 192.168.206.105@tcp101 [8/256/0/180] [ 7506.031419] LNet: Added LNI 192.168.206.105@tcp102 [8/256/0/180] [ 7506.387972] LNet: Added LNI 192.168.206.105@tcp103 [8/256/0/180] [ 7506.751993] LNet: Added LNI 192.168.206.105@tcp104 [8/256/0/180] [ 7507.126059] LNet: Added LNI 192.168.206.105@tcp105 [8/256/0/180] [ 7507.535462] LNet: Added LNI 192.168.206.105@tcp106 [8/256/0/180] [ 7507.935086] LNet: Added LNI 192.168.206.105@tcp107 [8/256/0/180] [ 7508.290261] LNet: Added LNI 192.168.206.105@tcp108 [8/256/0/180] [ 7508.627358] LNet: Added LNI 192.168.206.105@tcp109 [8/256/0/180] [ 7508.945288] LNet: Added LNI 192.168.206.105@tcp110 [8/256/0/180] [ 7509.268533] LNet: Added LNI 192.168.206.105@tcp111 [8/256/0/180] [ 7509.599356] LNet: Added LNI 192.168.206.105@tcp112 [8/256/0/180] [ 7509.930259] LNet: Added LNI 192.168.206.105@tcp113 [8/256/0/180] [ 7510.281968] LNet: Added LNI 192.168.206.105@tcp114 [8/256/0/180] [ 7510.600908] LNet: Added LNI 192.168.206.105@tcp115 [8/256/0/180] [ 7510.916849] LNet: Added LNI 192.168.206.105@tcp116 [8/256/0/180] [ 7511.227995] LNet: Added LNI 192.168.206.105@tcp117 [8/256/0/180] [ 7511.553343] LNet: Added LNI 192.168.206.105@tcp118 [8/256/0/180] [ 7511.857945] LNet: Added LNI 192.168.206.105@tcp119 [8/256/0/180] [ 7512.157466] LNet: Added LNI 192.168.206.105@tcp120 [8/256/0/180] [ 7512.498764] LNet: Added LNI 192.168.206.105@tcp121 [8/256/0/180] [ 7512.844810] LNet: Added LNI 192.168.206.105@tcp122 [8/256/0/180] [ 7513.187377] LNet: Added LNI 192.168.206.105@tcp123 [8/256/0/180] [ 7513.549605] LNet: Added LNI 192.168.206.105@tcp124 [8/256/0/180] [ 7513.882549] LNet: Added LNI 192.168.206.105@tcp125 [8/256/0/180] [ 7514.215495] LNet: Added LNI 192.168.206.105@tcp126 [8/256/0/180] [ 7514.570731] LNet: Added LNI 192.168.206.105@tcp127 [8/256/0/180] [ 7514.902524] LNet: Added LNI 192.168.206.105@tcp128 [8/256/0/180] [ 7515.245121] LNet: Added LNI 192.168.206.105@tcp129 [8/256/0/180] [ 7515.568592] LNet: Added LNI 192.168.206.105@tcp130 [8/256/0/180] [ 7515.899599] LNet: Added LNI 192.168.206.105@tcp131 [8/256/0/180] [ 7516.232591] LNet: Added LNI 192.168.206.105@tcp132 [8/256/0/180] [ 7516.570252] LNet: Added LNI 192.168.206.105@tcp133 [8/256/0/180] [ 7516.893651] LNet: Added LNI 192.168.206.105@tcp134 [8/256/0/180] [ 7517.241245] LNet: Added LNI 192.168.206.105@tcp135 [8/256/0/180] [ 7519.954093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7519.960770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.009727] Lustre: 462602:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [ 7521.063749] LNetError: 460067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [ 7521.069231] LNetError: Protocol error connecting to 192.168.206.105@tcp116 at host 192.168.206.105:988: is it running a compatible version of Lustre? [ 7521.073787] LNetError: Skipped 1 previous similar message [ 7521.112829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7521.123145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7521.164546] Lustre: lustre-MDT0000: new disk, initializing [ 7521.189916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7521.195523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7522.083591] LNetError: 462624:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [ 7522.088578] LNetError: 462624:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7522.092786] LNetError: Protocol error connecting to 192.168.206.105@tcp109 at host 192.168.206.105:988: is it running a compatible version of Lustre? [ 7522.660631] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7527.348428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.411143] Lustre: 462618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7527.447466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7527.450503] Lustre: Skipped 1 previous similar message [ 7527.493188] Lustre: lustre-MDT0001: new disk, initializing [ 7527.508066] Lustre: 463423:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [ 7527.511504] Lustre: 463423:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7527.558332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7527.571962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7527.575781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7529.147671] Lustre: 463681:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [ 7529.151315] Lustre: 463681:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7529.167210] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7533.470252] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7535.711940] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7538.637711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7538.767949] Lustre: lustre-OST0000: new disk, initializing [ 7538.770221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7538.773535] Lustre: 464593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7538.810831] Lustre: 464598:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [ 7538.814898] Lustre: 464598:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7538.836133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.147264] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7544.819152] Lustre: 465254:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.105@tcp: can't add NID 192.168.206.105@tcp130: rc = -75 [ 7544.822596] Lustre: 465254:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7544.840777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7544.844238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7544.886565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7545.114139] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7549.922031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7549.924618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7549.925462] LustreError: Skipped 1 previous similar message [ 7549.931683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7549.934688] Lustre: Skipped 1 previous similar message [ 7552.034146] LNetError: 460066:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [ 7552.041494] LNetError: Protocol error connecting to 192.168.206.105@tcp118 at host 192.168.206.105:988: is it running a compatible version of Lustre? [ 7553.830346] Lustre: server umount lustre-OST0000 complete [ 7555.105793] LNetError: 460067:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [ 7555.111052] LNetError: Protocol error connecting to 192.168.206.105@tcp116 at host 192.168.206.105:988: is it running a compatible version of Lustre? [ 7558.625031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7558.627322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7558.631552] Lustre: Skipped 1 previous similar message [ 7558.633828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7559.200854] LNetError: 460068:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.105 [ 7559.204479] LNetError: Protocol error connecting to 192.168.206.105@tcp109 at host 192.168.206.105:988: is it running a compatible version of Lustre? [ 7560.161724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7560.162805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.444287] Lustre: server umount lustre-MDT0000 complete [ 7562.757665] LustreError: 463422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872716 with bad export cookie 4306294392107449907 [ 7562.761128] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7562.761925] LustreError: 463422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7562.873843] Lustre: server umount lustre-MDT0001 complete [ 7566.619669] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7567.718919] Key type lgssc unregistered [ 7567.881028] LNet: 466501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7567.883848] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7567.886132] LNet: Removed LNI 192.168.206.105@tcp [ 7568.928158] LNet: Removed LNI 192.168.206.105@tcp100 [ 7568.930383] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7568.934300] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7568.937476] LNet: Removed LNI 192.168.206.105@tcp101 [ 7569.952225] LNet: Removed LNI 192.168.206.105@tcp102 [ 7569.953506] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7569.955540] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7569.958609] LNet: Removed LNI 192.168.206.105@tcp103 [ 7570.976121] LNet: Removed LNI 192.168.206.105@tcp104 [ 7572.000160] LNet: Removed LNI 192.168.206.105@tcp105 [ 7572.001714] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7572.004331] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7572.007242] LNet: Removed LNI 192.168.206.105@tcp106 [ 7573.024121] LNet: Removed LNI 192.168.206.105@tcp107 [ 7574.048143] LNet: Removed LNI 192.168.206.105@tcp108 [ 7575.072134] LNet: Removed LNI 192.168.206.105@tcp109 [ 7576.096104] LNet: Removed LNI 192.168.206.105@tcp110 [ 7576.097966] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7576.101132] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7576.104347] LNet: Removed LNI 192.168.206.105@tcp111 [ 7577.120388] LNet: Removed LNI 192.168.206.105@tcp112 [ 7578.144319] LNet: Removed LNI 192.168.206.105@tcp113 [ 7579.168177] LNet: Removed LNI 192.168.206.105@tcp114 [ 7580.192156] LNet: Removed LNI 192.168.206.105@tcp115 [ 7581.216245] LNet: Removed LNI 192.168.206.105@tcp116 [ 7582.240201] LNet: Removed LNI 192.168.206.105@tcp117 [ 7583.264099] LNet: Removed LNI 192.168.206.105@tcp118 [ 7584.288133] LNet: Removed LNI 192.168.206.105@tcp119 [ 7584.289567] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7584.291810] LNetError: 466501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7584.294077] LNet: Removed LNI 192.168.206.105@tcp120 [ 7584.295042] LNet: Removed LNI 192.168.206.105@tcp121 [ 7584.296021] LNet: Removed LNI 192.168.206.105@tcp122 [ 7584.296898] LNet: Removed LNI 192.168.206.105@tcp123 [ 7584.297978] LNet: Removed LNI 192.168.206.105@tcp124 [ 7584.299784] LNet: Removed LNI 192.168.206.105@tcp125 [ 7584.301061] LNet: Removed LNI 192.168.206.105@tcp126 [ 7584.302031] LNet: Removed LNI 192.168.206.105@tcp127 [ 7584.303251] LNet: Removed LNI 192.168.206.105@tcp128 [ 7584.304635] LNet: Removed LNI 192.168.206.105@tcp129 [ 7584.306342] LNet: Removed LNI 192.168.206.105@tcp130 [ 7584.307496] LNet: Removed LNI 192.168.206.105@tcp131 [ 7584.308749] LNet: Removed LNI 192.168.206.105@tcp132 [ 7584.310433] LNet: Removed LNI 192.168.206.105@tcp133 [ 7584.311397] LNet: Removed LNI 192.168.206.105@tcp134 [ 7584.318458] LNet: Removed LNI 192.168.206.105@tcp135 [ 7584.594734] Key type .llcrypt unregistered [ 7584.596339] Key type ._llcrypt unregistered [ 7592.659706] Key type ._llcrypt registered [ 7592.660840] Key type .llcrypt registered [ 7592.702104] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 7599.371762] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7599.715955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7599.739322] alg: No test for adler32 (adler32-zlib) [ 7600.599326] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 7600.703950] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7602.296167] Key type lgssc registered [ 7602.742298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7615.542276] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 09:06:08 (1788872768) [ 7622.019859] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7623.151848] Key type lgssc unregistered [ 7623.296364] LNet: 470553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7623.298893] LNetError: 470553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7623.309687] LNet: Removed LNI 192.168.206.105@tcp [ 7623.631540] Key type .llcrypt unregistered [ 7623.633087] Key type ._llcrypt unregistered [ 7630.911060] Key type ._llcrypt registered [ 7630.911923] Key type .llcrypt registered [ 7630.951671] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7631.290441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7631.328079] alg: No test for adler32 (adler32-zlib) [ 7632.193472] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 7632.297271] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7633.888368] Key type lgssc registered [ 7634.334957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7638.010119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7638.033689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7639.150252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7639.170862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7639.217973] Lustre: lustre-MDT0000: new disk, initializing [ 7639.259296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7639.271825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7641.000259] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7645.997292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.042731] Lustre: 471741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7646.074247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7646.077036] Lustre: Skipped 1 previous similar message [ 7646.120515] Lustre: lustre-MDT0001: new disk, initializing [ 7646.157863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7646.177251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7646.182181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7647.780120] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7651.734962] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7653.635578] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7655.903840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7655.999249] Lustre: lustre-OST0000: new disk, initializing [ 7656.001152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7656.004318] Lustre: 473707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7656.032324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7657.953585] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7661.724783] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7663.052864] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing lnet_if_list [ 7663.107370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7663.111311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7663.163862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7664.881786] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7664.894535] LNet: Added LNI 192.168.206.105@tcp10 [8/256/0/180] [ 7667.305442] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7667.319777] LNet: Added LNI 192.168.206.105@tcp11 [8/256/0/180] [ 7668.836424] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7668.848194] LNet: Added LNI 192.168.206.105@tcp12 [8/256/0/180] [ 7670.502409] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7670.523095] LNet: Added LNI 192.168.206.105@tcp13 [8/256/0/180] [ 7672.148139] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7672.159149] LNet: Added LNI 192.168.206.105@tcp14 [8/256/0/180] [ 7673.847630] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7673.862421] LNet: Added LNI 192.168.206.105@tcp15 [8/256/0/180] [ 7675.592128] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7675.605742] LNet: Added LNI 192.168.206.105@tcp16 [8/256/0/180] [ 7677.426898] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7677.439502] LNet: Added LNI 192.168.206.105@tcp17 [8/256/0/180] [ 7679.419596] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7679.433389] LNet: Added LNI 192.168.206.105@tcp18 [8/256/0/180] [ 7681.239438] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7681.258903] LNet: Added LNI 192.168.206.105@tcp19 [8/256/0/180] [ 7683.013864] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7683.025942] LNet: Added LNI 192.168.206.105@tcp20 [8/256/0/180] [ 7684.884720] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7684.895524] LNet: Added LNI 192.168.206.105@tcp21 [8/256/0/180] [ 7686.846642] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7686.862192] LNet: Added LNI 192.168.206.105@tcp22 [8/256/0/180] [ 7689.042671] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7689.059588] LNet: Added LNI 192.168.206.105@tcp23 [8/256/0/180] [ 7690.937306] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7690.948755] LNet: Added LNI 192.168.206.105@tcp24 [8/256/0/180] [ 7692.910261] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7692.924742] LNet: Added LNI 192.168.206.105@tcp25 [8/256/0/180] [ 7695.041277] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7695.059398] LNet: Added LNI 192.168.206.105@tcp26 [8/256/0/180] [ 7697.127551] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7697.139830] LNet: Added LNI 192.168.206.105@tcp27 [8/256/0/180] [ 7699.287590] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7699.300811] LNet: Added LNI 192.168.206.105@tcp28 [8/256/0/180] [ 7701.758413] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7701.772179] LNet: Added LNI 192.168.206.105@tcp29 [8/256/0/180] [ 7703.962983] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7703.978461] LNet: Added LNI 192.168.206.105@tcp30 [8/256/0/180] [ 7706.274944] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7706.289503] LNet: Added LNI 192.168.206.105@tcp31 [8/256/0/180] [ 7708.607128] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7708.622517] LNet: Added LNI 192.168.206.105@tcp32 [8/256/0/180] [ 7711.017923] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7711.028591] LNet: Added LNI 192.168.206.105@tcp33 [8/256/0/180] [ 7713.213833] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7713.223768] LNet: Added LNI 192.168.206.105@tcp34 [8/256/0/180] [ 7715.475326] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7715.484920] LNet: Added LNI 192.168.206.105@tcp35 [8/256/0/180] [ 7717.945809] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7717.956020] LNet: Added LNI 192.168.206.105@tcp36 [8/256/0/180] [ 7720.293900] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7720.307702] LNet: Added LNI 192.168.206.105@tcp37 [8/256/0/180] [ 7722.949174] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7722.966791] LNet: Added LNI 192.168.206.105@tcp38 [8/256/0/180] [ 7725.460442] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7725.474065] LNet: Added LNI 192.168.206.105@tcp39 [8/256/0/180] [ 7727.898809] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7727.914684] LNet: Added LNI 192.168.206.105@tcp40 [8/256/0/180] [ 7730.509434] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7730.524103] LNet: Added LNI 192.168.206.105@tcp41 [8/256/0/180] [ 7733.163453] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7733.176621] LNet: Added LNI 192.168.206.105@tcp42 [8/256/0/180] [ 7735.817164] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7735.830591] LNet: Added LNI 192.168.206.105@tcp43 [8/256/0/180] [ 7738.590624] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7738.602780] LNet: Added LNI 192.168.206.105@tcp44 [8/256/0/180] [ 7741.512208] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7741.527945] LNet: Added LNI 192.168.206.105@tcp45 [8/256/0/180] [ 7744.390820] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7744.405251] LNet: Added LNI 192.168.206.105@tcp46 [8/256/0/180] [ 7747.227344] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7747.244076] LNet: Added LNI 192.168.206.105@tcp47 [8/256/0/180] [ 7750.080830] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7750.094487] LNet: Added LNI 192.168.206.105@tcp48 [8/256/0/180] [ 7752.958418] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7752.973698] LNet: Added LNI 192.168.206.105@tcp49 [8/256/0/180] [ 7755.954981] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7755.967919] LNet: Added LNI 192.168.206.105@tcp50 [8/256/0/180] [ 7763.936809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7763.939271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7763.944452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7765.475533] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.475975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7765.482875] Lustre: Skipped 1 previous similar message [ 7765.485040] Lustre: Skipped 1 previous similar message [ 7769.887212] Lustre: server umount lustre-MDT0000 complete [ 7771.158116] LustreError: 473706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872924 with bad export cookie 4959974913296354922 [ 7771.161532] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.162349] LustreError: 473706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7771.282825] Lustre: server umount lustre-MDT0001 complete [ 7779.146815] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7782.641359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.072123] Lustre: 471012:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788872928/real 1788872928] req@000000001029094a x1875769069489408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788872944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7791.082785] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.096135] Lustre: 471013:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788872929/real 1788872929] req@00000000c09861f4 x1875769069489536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788872945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7796.193854] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x44d55ee791e0d689 [ 7796.197283] Lustre: MGC192.168.206.105@tcp: Connection restored to 0@lo (at 0@lo) [ 7796.301692] LustreError: 473699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7796.308334] LustreError: 473699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7796.337367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7797.344087] Lustre: 471010:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788872934/real 1788872934] req@0000000078e2dd12 x1875769069489664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788872950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7797.911342] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7800.354310] LustreError: 473700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7800.979610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7801.122086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7802.150069] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7802.651412] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7803.168113] Lustre: 471011:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788872939/real 1788872939] req@000000003126ed08 x1875769069489920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788872955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7805.592897] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7807.248108] Lustre: 471013:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788872944/real 1788872944] req@0000000072518c4b x1875769069490048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788872960 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7808.137879] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7817.700256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.705597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.708276] Lustre: Skipped 1 previous similar message [ 7822.305167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7822.308502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.314610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7823.392823] Lustre: server umount lustre-OST0000 complete [ 7826.914194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7826.918739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7826.921041] Lustre: Skipped 1 previous similar message [ 7838.177380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.180880] Lustre: Skipped 5 previous similar messages [ 7839.200112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7839.262772] Lustre: server umount lustre-MDT0000 complete [ 7840.777586] LustreError: 472533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872994 with bad export cookie 4959974913296357001 [ 7840.782784] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7840.783192] LustreError: 472533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7840.924263] Lustre: server umount lustre-MDT0001 complete [ 7845.454408] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7846.747378] Key type lgssc unregistered [ 7846.929602] LNet: 500669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7846.934701] LNetError: 500669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7846.937684] LNet: Removed LNI 192.168.206.105@tcp [ 7846.939245] LNet: Removed LNI 192.168.206.105@tcp10 [ 7846.940849] LNet: Removed LNI 192.168.206.105@tcp11 [ 7846.942161] LNet: Removed LNI 192.168.206.105@tcp12 [ 7846.943445] LNet: Removed LNI 192.168.206.105@tcp13 [ 7846.945667] LNet: Removed LNI 192.168.206.105@tcp14 [ 7846.946851] LNet: Removed LNI 192.168.206.105@tcp15 [ 7846.948631] LNet: Removed LNI 192.168.206.105@tcp16 [ 7846.950205] LNet: Removed LNI 192.168.206.105@tcp17 [ 7846.951791] LNet: Removed LNI 192.168.206.105@tcp18 [ 7846.954200] LNet: Removed LNI 192.168.206.105@tcp19 [ 7846.955707] LNet: Removed LNI 192.168.206.105@tcp20 [ 7846.957675] LNet: Removed LNI 192.168.206.105@tcp21 [ 7846.959091] LNet: Removed LNI 192.168.206.105@tcp22 [ 7846.960929] LNet: Removed LNI 192.168.206.105@tcp23 [ 7846.962257] LNet: Removed LNI 192.168.206.105@tcp24 [ 7846.964036] LNet: Removed LNI 192.168.206.105@tcp25 [ 7846.966794] LNet: Removed LNI 192.168.206.105@tcp26 [ 7846.972149] LNet: Removed LNI 192.168.206.105@tcp27 [ 7846.975521] LNet: Removed LNI 192.168.206.105@tcp28 [ 7846.978723] LNet: Removed LNI 192.168.206.105@tcp29 [ 7846.981801] LNet: Removed LNI 192.168.206.105@tcp30 [ 7846.986404] LNet: Removed LNI 192.168.206.105@tcp31 [ 7846.989296] LNet: Removed LNI 192.168.206.105@tcp32 [ 7846.991891] LNet: Removed LNI 192.168.206.105@tcp33 [ 7846.993674] LNet: Removed LNI 192.168.206.105@tcp34 [ 7846.995592] LNet: Removed LNI 192.168.206.105@tcp35 [ 7846.997383] LNet: Removed LNI 192.168.206.105@tcp36 [ 7846.999333] LNet: Removed LNI 192.168.206.105@tcp37 [ 7847.001040] LNet: Removed LNI 192.168.206.105@tcp38 [ 7847.003822] LNet: Removed LNI 192.168.206.105@tcp39 [ 7847.006502] LNet: Removed LNI 192.168.206.105@tcp40 [ 7847.008902] LNet: Removed LNI 192.168.206.105@tcp41 [ 7847.010930] LNet: Removed LNI 192.168.206.105@tcp42 [ 7847.012681] LNet: Removed LNI 192.168.206.105@tcp43 [ 7847.014392] LNet: Removed LNI 192.168.206.105@tcp44 [ 7847.016070] LNet: Removed LNI 192.168.206.105@tcp45 [ 7847.018218] LNet: Removed LNI 192.168.206.105@tcp46 [ 7847.020061] LNet: Removed LNI 192.168.206.105@tcp47 [ 7847.021236] LNet: Removed LNI 192.168.206.105@tcp48 [ 7847.022586] LNet: Removed LNI 192.168.206.105@tcp49 [ 7847.028488] LNet: Removed LNI 192.168.206.105@tcp50 [ 7847.351182] Key type .llcrypt unregistered [ 7847.352822] Key type ._llcrypt unregistered [ 7855.229180] Key type ._llcrypt registered [ 7855.230862] Key type .llcrypt registered [ 7855.284773] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 7862.726489] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7863.201496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7863.213389] alg: No test for adler32 (adler32-zlib) [ 7864.119215] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 7864.245355] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7865.842177] Key type lgssc registered [ 7866.366692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7881.133788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7881.143413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.231588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7882.245348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7882.285825] Lustre: lustre-MDT0000: new disk, initializing [ 7882.317235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7882.327085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7883.798177] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7888.200138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.259734] Lustre: 503835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7888.287136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7888.289391] Lustre: Skipped 1 previous similar message [ 7888.330831] Lustre: lustre-MDT0001: new disk, initializing [ 7888.376258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7888.393938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7888.398772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7889.885522] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7894.013701] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7896.325153] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7899.352563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7899.432479] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7899.466717] Lustre: lustre-OST0000: new disk, initializing [ 7899.468853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7899.471359] Lustre: 505848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7899.501853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7901.853876] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7905.870237] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7908.619970] Lustre: server umount lustre-OST0000 complete [ 7913.954043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.954335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7913.956345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7913.960507] Lustre: Skipped 1 previous similar message [ 7916.279740] Lustre: server umount lustre-MDT0000 complete [ 7917.943424] LustreError: 503826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873071 with bad export cookie 6151756460761787129 [ 7917.947425] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7917.947689] LustreError: 503826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7918.079068] Lustre: server umount lustre-MDT0001 complete [ 7922.402937] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 7923.614476] Key type lgssc unregistered [ 7923.773741] LNet: 507745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7923.776851] LNetError: 507745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7923.788983] LNet: Removed LNI 192.168.206.105@tcp [ 7924.185139] Key type .llcrypt unregistered [ 7924.186766] Key type ._llcrypt unregistered [ 7932.926946] Key type ._llcrypt registered [ 7932.928269] Key type .llcrypt registered [ 7932.977376] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 7933.380914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7933.403063] alg: No test for adler32 (adler32-zlib) [ 7934.280191] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 7934.378529] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 7935.968178] Key type lgssc registered [ 7936.491202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7939.849615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7939.857703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7941.075102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7942.834983] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7946.519062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7946.689631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7948.281895] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7951.354738] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7953.654292] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7956.799092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.004241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7959.695973] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7962.980890] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7966.277979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7966.283856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7966.300039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 7978.858232] loop0: detected capacity change from 0 to 204800000 [ 7978.939578] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7978.948187] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7978.956385] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7982.060978] loop0: detected capacity change from 0 to 204800000 [ 7982.149606] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7982.156955] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7982.176022] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7985.127796] loop0: detected capacity change from 0 to 204800000 [ 7988.154380] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7988.257487] Lustre: fs15246-OST0000: new disk, initializing [ 7988.259714] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7988.263554] Lustre: 512619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7988.300324] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.642931] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 7997.541758] loop1: detected capacity change from 0 to 204800000 [ 8000.414295] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.475852] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8000.516725] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8000.567626] Lustre: fs15246-MDT0000: new disk, initializing [ 8000.669373] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.690303] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8000.745139] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8001.761651] Lustre: 513738:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f22ee8b0 x1875769386109184/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788873171 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8002.861836] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8010.852100] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8010.855566] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8010.871321] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8012.260720] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8012.268085] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8017.380865] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8018.403863] Lustre: server umount fs15246-MDT0000 complete [ 8022.283572] Lustre: server umount fs15246-OST0000 complete [ 8027.617279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8027.620417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8027.628902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8027.631511] Lustre: Skipped 1 previous similar message [ 8032.741563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8032.744691] Lustre: Skipped 1 previous similar message [ 8033.049956] Lustre: server umount lustre-OST0000 complete [ 8037.857672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.858830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8037.863025] Lustre: Skipped 2 previous similar messages [ 8040.735343] Lustre: server umount lustre-MDT0000 complete [ 8042.192271] LustreError: 508984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873195 with bad export cookie 10225891904907428872 [ 8042.195490] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8042.197989] LustreError: 508984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8042.356131] Lustre: server umount lustre-MDT0001 complete [ 8046.736675] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 8047.903412] Key type lgssc unregistered [ 8048.035347] LNet: 516830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8048.038311] LNetError: 516830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8048.046965] LNet: Removed LNI 192.168.206.105@tcp [ 8048.414337] Key type .llcrypt unregistered [ 8048.415956] Key type ._llcrypt unregistered [ 8057.158852] Key type ._llcrypt registered [ 8057.161315] Key type .llcrypt registered [ 8057.216056] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 8057.600599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8057.635247] alg: No test for adler32 (adler32-zlib) [ 8058.517752] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 8058.624381] LNet: Added LNI 192.168.206.105@tcp [8/256/0/180] [ 8060.224746] Key type lgssc registered [ 8060.709784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8077.548381] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 8081.986920] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 8097.675789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8097.690990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8098.783234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8098.796228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8098.837287] Lustre: lustre-MDT0000: new disk, initializing [ 8098.866638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8098.873820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8100.449132] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8105.299075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.346883] Lustre: 521261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8105.371658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8105.373756] Lustre: Skipped 1 previous similar message [ 8105.415261] Lustre: lustre-MDT0001: new disk, initializing [ 8105.458857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8105.473106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8105.477270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8107.144170] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8111.537713] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8113.824189] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8116.626691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8116.756271] Lustre: lustre-OST0000: new disk, initializing [ 8116.761881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8116.766948] Lustre: 523228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8116.805518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8119.442413] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_default_debug -1 all [ 8124.041517] Lustre: DEBUG MARKER: oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8126.018520] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8127.006070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8127.016703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8127.063552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8127.170564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 8129.128932] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8129.237039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8132.071619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8132.071686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8132.078308] LustreError: Skipped 1 previous similar message [ 8132.085502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8132.090523] Lustre: Skipped 1 previous similar message [ 8136.289170] Lustre: server umount lustre-OST0000 complete [ 8141.282043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8141.285599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.292103] Lustre: Skipped 1 previous similar message [ 8141.295056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8142.307270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8142.307488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8142.318765] Lustre: Skipped 1 previous similar message [ 8144.102934] Lustre: server umount lustre-MDT0000 complete [ 8145.754595] LustreError: 521255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873299 with bad export cookie 8187033123579067412 [ 8145.758352] LustreError: MGC192.168.206.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8145.760210] LustreError: 521255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8145.907949] Lustre: server umount lustre-MDT0001 complete [ 8158.704459] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing set_hostid [ 8164.390768] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing load_modules_local [ 8179.026403] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8057 sec ========== 09:15:31 (1788873331) [ 8179.964704] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:15:32 (1788873332) === [ 8181.611581] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:15:34 (1788873334) === [ 8200.997626] Lustre: DEBUG MARKER: oleg605-server.virtnet: executing unload_modules_local [ 8202.219143] Key type lgssc unregistered [ 8202.381561] LNet: 529867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8202.386617] LNetError: 529867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8202.395891] LNet: Removed LNI 192.168.206.105@tcp [ 8202.736120] Key type .llcrypt unregistered [ 8202.737605] Key type ._llcrypt unregistered