[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 852884267 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002367] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007361] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012030] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.019762] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023412] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026100] *** VALIDATE tmpfs *** [ 0.028379] *** VALIDATE proc *** [ 0.029222] *** VALIDATE cgroup *** [ 0.030008] *** VALIDATE cgroup2 *** [ 0.031249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034026] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.038191] debug: unmapping init [mem 0xffffffffaba59000-0xffffffffaba60fff] [ 0.040248] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041680] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042030] ... version: 2 [ 0.043010] ... bit width: 48 [ 0.044007] ... generic registers: 4 [ 0.045011] ... value mask: 0000ffffffffffff [ 0.046010] ... max period: 00007fffffffffff [ 0.047012] ... fixed-purpose events: 3 [ 0.048008] ... event mask: 000000070000000f [ 0.049302] rcu: Hierarchical SRCU implementation. [ 0.051575] smp: Bringing up secondary CPUs ... [ 0.052604] x86: Booting SMP configuration: [ 0.053019] .... node #0, CPUs: #1 #2 #3 [ 0.059127] smp: Brought up 1 node, 4 CPUs [ 0.061019] smpboot: Max logical packages: 1 [ 0.062013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.095021] node 0 deferred pages initialised in 31ms [ 0.099507] devtmpfs: initialized [ 0.100381] x86/mm: Memory block size: 128MB [ 0.103613] gcov: version magic: 0x41383552 [ 0.105311] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106184] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.107431] pinctrl core: initialized pinctrl subsystem [ 0.108222] [ 0.108734] ************************************************************* [ 0.109014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.110013] ** ** [ 0.111012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.112015] ** ** [ 0.113013] ** This means that this kernel is built to expose internal ** [ 0.114010] ** IOMMU data structures, which may compromise security on ** [ 0.115007] ** your system. ** [ 0.116009] ** ** [ 0.117012] ** If you see this message and you are not debugging the ** [ 0.118011] ** kernel, report this immediately to your vendor! ** [ 0.119013] ** ** [ 0.120019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121011] ************************************************************* [ 0.122802] NET: Registered protocol family 16 [ 0.123747] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.124065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.125064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.126703] cpuidle: using governor menu [ 0.130264] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.133469] PCI: Using configuration type 1 for base access [ 0.136129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.143228] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.144033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.147128] cryptd: max_cpu_qlen set to 1000 [ 0.149000] ACPI: Added _OSI(Module Device) [ 0.149015] ACPI: Added _OSI(Processor Device) [ 0.150000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.153014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.159998] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.168791] ACPI: Interpreter enabled [ 0.169060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.170022] ACPI: Using IOAPIC for interrupt routing [ 0.171169] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.172438] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.185293] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.187038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.191019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.195082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.201638] acpiphp: Slot [2] registered [ 0.203110] acpiphp: Slot [5] registered [ 0.205198] acpiphp: Slot [6] registered [ 0.206110] acpiphp: Slot [7] registered [ 0.208141] acpiphp: Slot [8] registered [ 0.210153] acpiphp: Slot [9] registered [ 0.211123] acpiphp: Slot [10] registered [ 0.213232] acpiphp: Slot [3] registered [ 0.215299] acpiphp: Slot [4] registered [ 0.217213] acpiphp: Slot [11] registered [ 0.219096] acpiphp: Slot [12] registered [ 0.221408] acpiphp: Slot [13] registered [ 0.224122] acpiphp: Slot [14] registered [ 0.225312] acpiphp: Slot [15] registered [ 0.228118] acpiphp: Slot [16] registered [ 0.230148] acpiphp: Slot [17] registered [ 0.232114] acpiphp: Slot [18] registered [ 0.234106] acpiphp: Slot [19] registered [ 0.235092] acpiphp: Slot [20] registered [ 0.237188] acpiphp: Slot [21] registered [ 0.239200] acpiphp: Slot [22] registered [ 0.241142] acpiphp: Slot [23] registered [ 0.243120] acpiphp: Slot [24] registered [ 0.244102] acpiphp: Slot [25] registered [ 0.245212] acpiphp: Slot [26] registered [ 0.247109] acpiphp: Slot [27] registered [ 0.249129] acpiphp: Slot [28] registered [ 0.251094] acpiphp: Slot [29] registered [ 0.252217] acpiphp: Slot [30] registered [ 0.254152] acpiphp: Slot [31] registered [ 0.256064] PCI host bridge to bus 0000:00 [ 0.258085] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.261021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.265021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.269036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.274033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.277103] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.280378] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.285530] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.290535] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.302988] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.309060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.312019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.315021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.319031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.323712] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327184] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.331043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.335038] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.342017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.357014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.362014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.369168] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.384024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.397030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.489017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.515962] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.527146] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.538018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.561027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.626428] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.642016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.668016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.756019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.858845] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.869022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.924023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.941022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.953414] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.962024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.969024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.987025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.003742] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.004000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.027018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.042016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.055696] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.058546] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.061437] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.067591] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.070311] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.076217] iommu: Default domain type: Passthrough [ 1.077314] SCSI subsystem initialized [ 1.078000] ACPI: bus type USB registered [ 1.080120] usbcore: registered new interface driver usbfs [ 1.083091] usbcore: registered new interface driver hub [ 1.085105] usbcore: registered new device driver usb [ 1.088203] pps_core: LinuxPPS API ver. 1 registered [ 1.091013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.095072] PTP clock support registered [ 1.097198] EDAC MC: Ver: 3.0.0 [ 1.100202] PCI: Using ACPI for IRQ routing [ 1.103169] NetLabel: Initializing [ 1.105012] NetLabel: domain hash size = 128 [ 1.107050] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.110146] NetLabel: unlabeled traffic allowed by default [ 1.113005] vgaarb: loaded [ 1.115133] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.118012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.124377] clocksource: Switched to clocksource kvm-clock [ 1.173009] hrtimer: interrupt took 5351861 ns [ 1.495424] VFS: Disk quotas dquot_6.6.0 [ 1.498410] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.502794] *** VALIDATE ramfs *** [ 1.504800] *** VALIDATE hugetlbfs *** [ 1.507035] pnp: PnP ACPI init [ 1.510710] pnp: PnP ACPI: found 6 devices [ 1.532598] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.536282] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.540912] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.546197] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.604714] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.610050] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.619262] NET: Registered protocol family 2 [ 1.627179] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.639419] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.644715] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.650429] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.654278] TCP: Hash tables configured (established 65536 bind 65536) [ 1.657485] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.663512] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.668511] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.675763] NET: Registered protocol family 1 [ 1.683069] RPC: Registered named UNIX socket transport module. [ 1.688415] RPC: Registered udp transport module. [ 1.691136] RPC: Registered tcp transport module. [ 1.694455] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.699574] NET: Registered protocol family 44 [ 1.710089] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.723779] pci 0000:00:00.0: quirk_natoma+0x0/0x20 took 13368 usecs [ 1.726373] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.728748] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.731767] PCI: CLS 0 bytes, default 64 [ 1.733811] Unpacking initramfs... [ 3.962761] debug: unmapping init [mem 0xffff9b123cc54000-0xffff9b123ffbffff] [ 3.965441] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.966918] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.968677] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.505744] Initialise system trusted keyrings [ 4.507322] Key type blacklist registered [ 4.572662] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.639688] zbud: loaded [ 4.643745] *** VALIDATE nfs *** [ 4.645381] *** VALIDATE nfs4 *** [ 4.647340] pstore: using deflate compression [ 4.652309] Platform Keyring initialized [ 4.809300] NET: Registered protocol family 38 [ 4.811285] Key type asymmetric registered [ 4.813588] Asymmetric key parser 'x509' registered [ 4.815862] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.819863] io scheduler mq-deadline registered [ 4.821522] io scheduler kyber registered [ 4.822996] io scheduler bfq registered [ 4.824769] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.829881] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.834617] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.838665] ACPI: Power Button [PWRF] [ 4.845703] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.854442] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.888463] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.899319] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.935408] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.964714] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.995323] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.000476] Non-volatile memory driver v1.3 [ 5.002650] Linux agpgart interface v0.103 [ 5.050945] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.053726] vda: detected capacity change from 0 to 74698752 [ 5.083433] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.087259] vdb: detected capacity change from 0 to 1073741824 [ 5.104178] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.107104] vdc: detected capacity change from 0 to 2621440000 [ 5.124925] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.128294] vdd: detected capacity change from 0 to 2621440000 [ 5.163301] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.166459] vde: detected capacity change from 0 to 4294967296 [ 5.187498] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.191161] vdf: detected capacity change from 0 to 4294967296 [ 5.240381] libphy: Fixed MDIO Bus: probed [ 5.265846] usbcore: registered new interface driver usbserial_generic [ 5.268493] usbserial: USB Serial support registered for generic [ 5.274177] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.282088] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.285329] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.291461] mousedev: PS/2 mouse device common for all mice [ 5.296364] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.299683] rtc_cmos 00:05: RTC can wake from S4 [ 5.312104] rtc_cmos 00:05: registered as rtc0 [ 5.317338] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.323223] intel_pstate: CPU model not supported [ 5.330451] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.335985] hid: raw HID events driver (C) Jiri Kosina [ 5.347026] usbcore: registered new interface driver usbhid [ 5.351323] usbhid: USB HID core driver [ 5.354171] drop_monitor: Initializing network drop monitor service [ 5.358731] Initializing XFRM netlink socket [ 5.362504] NET: Registered protocol family 10 [ 5.364535] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.374342] Segment Routing with IPv6 [ 5.382974] NET: Registered protocol family 17 [ 5.385190] mpls_gso: MPLS GSO support [ 5.395322] RAS: Correctable Errors collector initialized. [ 5.398455] AVX version of gcm_enc/dec engaged. [ 5.401202] AES CTR mode by8 optimization enabled [ 5.695331] sched_clock: Marking stable (5695275925, 0)->(6982143220, -1286867295) [ 5.698378] registered taskstats version 1 [ 5.699707] Loading compiled-in X.509 certificates [ 5.701325] zswap: loaded using pool lzo/zbud [ 5.723550] Key type big_key registered [ 5.734175] Key type encrypted registered [ 5.735432] ima: No TPM chip found, activating TPM-bypass! [ 5.736750] ima: Allocated hash algorithm: sha1 [ 5.738782] ima: No architecture policies found [ 5.740775] evm: Initialising EVM extended attributes: [ 5.742983] evm: security.selinux [ 5.744309] evm: security.ima [ 5.745643] evm: security.capability [ 5.747353] evm: HMAC attrs: 0x1 [ 5.750347] rtc_cmos 00:05: setting system clock to 2026-08-10 08:52:52 UTC (1786351972) [ 5.758661] debug: unmapping init [mem 0xffffffffaca03000-0xffffffffacbfffff] [ 5.761374] debug: unmapping init [mem 0xffffffffab782000-0xffffffffaba58fff] [ 5.769598] Write protecting the kernel read-only data: 28672k [ 5.777071] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 5.781108] debug: unmapping init [mem 0xffffffffaa714000-0xffffffffaa7fffff] [ 5.832309] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.842634] systemd[1]: Detected virtualization kvm. [ 5.844805] systemd[1]: Detected architecture x86-64. [ 5.847250] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.876661] systemd[1]: No hostname configured. [ 5.878598] systemd[1]: Set hostname to . [ 5.880830] random: systemd: uninitialized urandom read (16 bytes read) [ 5.883238] systemd[1]: Initializing machine ID from random generator. [ 6.019300] random: ln: uninitialized urandom read (6 bytes read) [ 6.187500] random: systemd: uninitialized urandom read (16 bytes read) [ 6.190605] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.206331] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 6.239263] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.106642] device-mapper: uevent: version 1.0.3 [ 7.109227] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.570828] virtio_net virtio0 ens2: renamed from eth0 [ 9.593960] random: fast init done [ 10.049714] scsi host0: ata_piix [ 10.301173] scsi host1: ata_piix [ 10.305452] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.312111] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.254353] random: crng init done [ 16.256652] random: 7 urandom warning(s) missed due to ratelimiting [ 18.428280] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.279776] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.836187] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.884694] SELinux: Disabled at runtime. [ 24.999649] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.034601] systemd[1]: Detected virtualization kvm. [ 25.041270] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.195643] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.206945] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.224609] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.232092] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.239647] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.265246] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.282574] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Pro[ 27.480053] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS cess Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 29.110407] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.139730] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.340671] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.952909] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.993467] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.328406] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.361583] NFS: Registering the id_resolver key type [ 37.369739] Key type id_resolver registered [ 37.377522] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg138-server login: [ 85.027043] libcfs: loading out-of-tree module taints kernel. [ 85.062547] Key type ._llcrypt registered [ 85.064521] Key type .llcrypt registered [ 85.141380] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 96.865734] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 97.981674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.993535] alg: No test for adler32 (adler32-zlib) [ 99.092510] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 99.543722] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 101.175229] Key type lgssc registered [ 102.176665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.942584] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 137.048640] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 145.324713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.351148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.555532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.580822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.638640] Lustre: lustre-MDT0000: new disk, initializing [ 146.710994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.721632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 149.256559] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.541497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.631656] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.656484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.660072] Lustre: Skipped 1 previous similar message [ 157.716580] Lustre: lustre-MDT0001: new disk, initializing [ 157.764224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.781155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.785993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 160.819385] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.407285] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 170.815711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.065971] Lustre: lustre-OST0000: new disk, initializing [ 171.069465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.075183] Lustre: 8417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 171.124688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.349755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.358121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 171.421563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 175.548242] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.348238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.458508] Lustre: lustre-OST0001: new disk, initializing [ 186.460507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 186.463589] Lustre: 9491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.518716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.756666] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.608881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 196.618462] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 196.676063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 201.572188] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.745374] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 209.394423] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing check_logdir /tmp/testlogs/ [ 212.472528] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing yml_node [ 216.531462] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 218.552154] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 220.651183] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 221.924950] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 04:56:26 EDT 2026 [ 234.695722] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 235.919413] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 249.827755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 249.834639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 249.847870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 252.903095] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.903883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 252.909979] Lustre: Skipped 2 previous similar messages [ 252.920316] Lustre: Skipped 2 previous similar messages [ 255.695969] Lustre: server umount lustre-MDT0000 complete [ 257.857802] LustreError: 8416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786352224 with bad export cookie 5126287345294728893 [ 257.859600] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 257.863518] LustreError: 8416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 258.026093] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.031828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 258.031836] Lustre: Skipped 1 previous similar message [ 262.048513] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 262.053048] Lustre: Skipped 2 previous similar messages [ 267.233435] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 267.233507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 267.245651] LustreError: 6488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 267.258509] Lustre: Skipped 1 previous similar message [ 268.256412] LustreError: 12432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 268.263384] LustreError: 12432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 272.351147] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 272.490478] Lustre: server umount lustre-MDT0001 complete [ 274.471060] Lustre: server umount lustre-OST0000 complete [ 276.453156] Lustre: server umount lustre-OST0001 complete [ 279.390408] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 284.969194] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 307.158730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 307.288774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 307.308928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 307.357392] Lustre: lustre-MDT0000: new disk, initializing [ 307.399829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.409447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 310.112091] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 317.134089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 317.205971] Lustre: 15040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 317.242249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 317.246434] Lustre: Skipped 1 previous similar message [ 317.291244] Lustre: lustre-MDT0001: new disk, initializing [ 317.337119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 317.353533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 317.358737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 319.930326] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 326.771809] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 330.376876] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 334.458411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 334.641475] Lustre: lustre-OST0000: new disk, initializing [ 334.643707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 334.647336] Lustre: 17008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 334.714123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 336.338040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 336.342923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 336.414697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 338.249957] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 344.309515] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 347.161117] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 347.266275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 349.578764] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 349.693851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 351.715268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 351.718749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.724242] LustreError: Skipped 1 previous similar message [ 351.726939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 351.734247] Lustre: Skipped 1 previous similar message [ 351.736993] Lustre: Skipped 2 previous similar messages [ 365.023115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 365.130803] Lustre: server umount lustre-OST0000 complete [ 368.609587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 368.614174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.619404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.622211] Lustre: Skipped 4 previous similar messages [ 373.305334] Lustre: server umount lustre-MDT0000 complete [ 375.473618] LustreError: 15031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786352342 with bad export cookie 5126287345294730426 [ 375.476816] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.483100] LustreError: 15031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 375.689438] Lustre: server umount lustre-MDT0001 complete [ 378.008626] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:59:02 (1786352342) [ 379.166561] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 380.260972] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:59:04 (1786352344) [ 384.278486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.569116] LustreError: 19414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.620106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.503542] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 390.115057] LustreError: 19415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.881731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.539703] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 399.884351] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 403.210766] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 407.453519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 411.033859] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 414.756598] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 423.396979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 423.398641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.401877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 423.407440] Lustre: Skipped 2 previous similar messages [ 423.417089] Lustre: Skipped 3 previous similar messages [ 424.900769] Lustre: server umount lustre-OST0000 complete [ 433.468539] Lustre: server umount lustre-MDT0000 complete [ 433.632828] LustreError: 19415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.641269] LustreError: 19415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 435.549258] LustreError: 19397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786352402 with bad export cookie 5126287345294731301 [ 435.550447] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 435.559787] LustreError: 19397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 435.707604] Lustre: server umount lustre-MDT0001 complete [ 441.618535] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 443.293928] Key type lgssc unregistered [ 443.487606] LNet: 22946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.495497] LNetError: 22946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 443.505542] LNet: Removed LNI 192.168.201.138@tcp [ 444.057119] Key type .llcrypt unregistered [ 444.058610] Key type ._llcrypt unregistered [ 457.808678] Key type ._llcrypt registered [ 457.810529] Key type .llcrypt registered [ 457.879393] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 458.436508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 458.490050] alg: No test for adler32 (adler32-zlib) [ 459.482713] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 459.660996] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 461.287159] Key type lgssc registered [ 461.881590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 466.667555] loop: module loaded [ 466.730654] loop0: detected capacity change from 0 to 204800000 [ 466.858845] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 466.867947] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 467.053920] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 470.506574] loop0: detected capacity change from 0 to 204800000 [ 470.663746] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 470.669799] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 470.677911] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 474.700614] loop0: detected capacity change from 0 to 204800000 [ 478.208889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 478.228629] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.362903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 479.392080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 479.434292] Lustre: lustre-MDT0000: new disk, initializing [ 479.477573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.493567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 481.975446] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 488.038217] loop1: detected capacity change from 0 to 204800000 [ 491.579394] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.782798] Lustre: lustre-OST07c6: new disk, initializing [ 491.784444] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 491.786381] Lustre: Skipped 1 previous similar message [ 491.790065] Lustre: 26106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 491.839237] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 492.092132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 492.098098] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 492.140229] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 496.185214] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 500.591762] Lustre: 26113:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 500.596478] Lustre: 26113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 500.600041] Lustre: 26113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 500.603980] Lustre: 26113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 500.608107] Lustre: 26113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 500.613156] Lustre: 26113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 515.908404] Lustre: 26113:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 515.915146] Lustre: 26113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 515.920304] Lustre: 26113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 515.932622] Lustre: 26113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 515.936790] Lustre: 26113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 515.940545] Lustre: 26113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 520.071794] Lustre: 25096:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 520.076126] Lustre: 25096:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 520.079384] Lustre: 25096:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 520.083127] Lustre: 25096:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 520.089211] Lustre: 25096:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 520.091649] Lustre: 25096:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 520.094164] Lustre: 25096:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 520.096933] Lustre: 25096:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 520.099085] Lustre: 25096:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 520.102158] Lustre: 25096:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 520.105808] Lustre: 25096:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 520.111163] Lustre: 25096:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 523.230203] Lustre: 25096:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 523.239700] Lustre: 25096:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 523.247422] Lustre: 25096:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 523.252800] Lustre: 25096:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 523.259183] Lustre: 25096:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 523.265095] Lustre: 25096:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 523.269826] Lustre: 25096:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 523.275166] Lustre: 25096:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 523.280954] Lustre: 25096:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 523.284283] Lustre: 25096:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 523.287827] Lustre: 25096:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 523.293896] Lustre: 25096:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 527.376244] Lustre: 26113:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 527.380588] Lustre: 26113:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 527.384218] Lustre: 26113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 527.387528] Lustre: 26113:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 527.391075] Lustre: 26113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 527.394505] Lustre: 26113:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 527.399768] Lustre: 26113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 527.405465] Lustre: 26113:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 527.411909] Lustre: 26113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 527.414375] Lustre: 26113:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 527.416760] Lustre: 26113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 527.418855] Lustre: 26113:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.687931] Lustre: 26113:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 537.696057] Lustre: 26113:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 537.703431] Lustre: 26113:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 537.711086] Lustre: 26113:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 537.717071] Lustre: 26113:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 537.720621] Lustre: 26113:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 537.728919] Lustre: 26113:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 537.733688] Lustre: 26113:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 537.738155] Lustre: 26113:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 537.742213] Lustre: 26113:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 537.746790] Lustre: 26113:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 537.750283] Lustre: 26113:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 584.165067] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.165751] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 584.171955] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 584.868991] Lustre: server umount lustre-OST07c6 complete [ 590.507630] Lustre: server umount lustre-MDT0000 complete [ 600.918402] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 602.580540] Key type lgssc unregistered [ 602.776618] LNet: 28693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.779788] LNetError: 28693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.794193] LNet: Removed LNI 192.168.201.138@tcp [ 603.278354] Key type .llcrypt unregistered [ 603.281109] Key type ._llcrypt unregistered [ 614.805362] Key type ._llcrypt registered [ 614.807265] Key type .llcrypt registered [ 614.870643] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 615.854535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 615.882582] alg: No test for adler32 (adler32-zlib) [ 616.792843] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 616.974032] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 618.615203] Key type lgssc registered [ 619.349750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.421125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 624.440595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.705758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.094646] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 630.754180] LustreError: 29942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.766022] LustreError: 29942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 633.108646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.353680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 635.857506] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 640.769625] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 644.749763] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 649.725758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 649.915913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.929076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 653.973859] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 658.982921] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 669.664791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 669.672560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.685339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 669.695174] Lustre: Skipped 1 previous similar message [ 669.702635] Lustre: Skipped 1 previous similar message [ 671.489378] Lustre: server umount lustre-OST0000 complete [ 674.272964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 674.279072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.294169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 679.907705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 679.914142] Lustre: Skipped 2 previous similar messages [ 680.150337] Lustre: server umount lustre-MDT0000 complete [ 682.320905] LustreError: 29922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786352649 with bad export cookie 15144280050459452744 [ 682.330229] LustreError: 29922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 682.330259] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.542739] Lustre: server umount lustre-MDT0001 complete [ 689.229879] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 690.930511] Key type lgssc unregistered [ 691.153697] LNet: 33676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.161717] LNetError: 33676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.174146] LNet: Removed LNI 192.168.201.138@tcp [ 691.744542] Key type .llcrypt unregistered [ 691.748185] Key type ._llcrypt unregistered [ 702.645799] Key type ._llcrypt registered [ 702.646841] Key type .llcrypt registered [ 702.715847] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 703.191511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 703.245486] alg: No test for adler32 (adler32-zlib) [ 704.214637] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 704.355317] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 705.967158] Key type lgssc registered [ 706.707726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 711.858520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 711.879132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.164846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.844124] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 721.481654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.699499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 724.388129] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 729.387176] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 732.859033] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 737.314171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.507191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.540505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 741.622257] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 746.788590] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 757.217713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 757.221797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.222414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 757.228350] Lustre: Skipped 1 previous similar message [ 761.705912] Lustre: server umount lustre-OST0000 complete [ 767.467710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.475461] Lustre: Skipped 1 previous similar message [ 767.480966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.488435] Lustre: Skipped 2 previous similar messages [ 770.770755] Lustre: server umount lustre-MDT0000 complete [ 772.582240] LustreError: 36164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.594550] LustreError: 36164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 773.725975] LustreError: 34905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786352740 with bad export cookie 5734150112824412937 [ 773.741832] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 773.745790] LustreError: 34905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 774.008079] Lustre: server umount lustre-MDT0001 complete [ 782.043338] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 784.097351] Key type lgssc unregistered [ 784.322639] LNet: 38459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.327619] LNetError: 38459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.391544] LNet: Removed LNI 192.168.201.138@tcp [ 786.063616] Key type .llcrypt unregistered [ 786.074709] Key type ._llcrypt unregistered [ 802.383256] Key type ._llcrypt registered [ 802.385244] Key type .llcrypt registered [ 802.466770] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 803.484392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 803.510378] alg: No test for adler32 (adler32-zlib) [ 804.471117] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 804.657614] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 806.287271] Key type lgssc registered [ 806.985818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.647171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 811.661047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.937903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.686088] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 821.027614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.241638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.809963] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 828.853294] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 832.947593] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 837.391687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 837.601441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.643711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 841.178481] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 845.567141] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 850.914556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.915237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 850.920705] Lustre: Skipped 1 previous similar message [ 850.926916] Lustre: Skipped 2 previous similar messages [ 853.013365] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 856.047530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.054144] Lustre: Skipped 2 previous similar messages [ 856.847592] LustreError: 42427:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000b44afb02) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 856.934820] Lustre: server umount lustre-MDT0000 complete [ 858.102973] LustreError: 39847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.122500] LustreError: 39847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 859.477081] LustreError: 39833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786352826 with bad export cookie 939112360506462249 [ 859.478349] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 859.483589] LustreError: 39833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 861.154994] LustreError: 41782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.159571] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.177872] Lustre: Skipped 1 previous similar message [ 861.180751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 863.225546] LustreError: 39850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.767322] Lustre: server umount lustre-MDT0001 complete [ 885.543418] Lustre: server umount lustre-OST0000 complete [ 896.710822] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 898.452887] Key type lgssc unregistered [ 898.642275] LNet: 43728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.649091] LNetError: 43728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.672744] LNet: Removed LNI 192.168.201.138@tcp [ 899.271158] Key type .llcrypt unregistered [ 899.272929] Key type ._llcrypt unregistered [ 914.850534] Key type ._llcrypt registered [ 914.851643] Key type .llcrypt registered [ 914.943335] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 915.550891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.618036] alg: No test for adler32 (adler32-zlib) [ 916.575755] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 916.745565] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 918.375167] Key type lgssc registered [ 919.249325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.281127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 924.293060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.586876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.992337] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 933.367650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.614538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 936.303353] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 940.703652] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 944.567452] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 949.636832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 949.863723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.907144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 953.869777] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 958.641957] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 964.578920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.581812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.588306] Lustre: Skipped 1 previous similar message [ 964.596382] Lustre: Skipped 1 previous similar message [ 966.165712] Lustre: lustre-OST0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 969.699260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.702577] Lustre: Skipped 1 previous similar message [ 974.819857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.822464] Lustre: Skipped 1 previous similar message [ 977.887093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 977.986937] Lustre: server umount lustre-OST0000 complete [ 995.302256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.302262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 995.305209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.316621] Lustre: Skipped 1 previous similar message [ 995.328029] Lustre: Skipped 2 previous similar messages [ 998.790260] Lustre: server umount lustre-MDT0000 complete [ 1000.421716] LustreError: 45120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.434832] LustreError: 45120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1001.063235] LustreError: 47699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786352967 with bad export cookie 14159518540152739717 [ 1001.072539] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.074053] LustreError: 47699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1007.351957] Lustre: server umount lustre-MDT0001 complete [ 1013.558825] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1015.282662] Key type lgssc unregistered [ 1015.489772] LNet: 48839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.495788] LNetError: 48839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.507356] LNet: Removed LNI 192.168.201.138@tcp [ 1015.965304] Key type .llcrypt unregistered [ 1015.967159] Key type ._llcrypt unregistered [ 1030.015277] Key type ._llcrypt registered [ 1030.016457] Key type .llcrypt registered [ 1030.083476] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1030.716956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1030.751861] alg: No test for adler32 (adler32-zlib) [ 1031.724688] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1031.853587] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1033.463605] Key type lgssc registered [ 1034.180840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1038.967486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1038.977413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.230339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.678900] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1047.720158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.939764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1050.297773] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1054.361117] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1057.524215] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1061.416097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1061.612814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.590220] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1064.687266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1068.825368] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1076.060374] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1078.007294] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1089.193861] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-10 5h10m53s [ 1090.102917] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1093.605515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.610624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.613215] Lustre: Skipped 2 previous similar messages [ 1093.622502] Lustre: Skipped 2 previous similar messages [ 1094.694471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1094.701568] Lustre: Skipped 2 previous similar messages [ 1096.694738] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 1096.698905] Lustre: Skipped 1 previous similar message [ 1097.464848] Lustre: server umount lustre-MDT0000 complete [ 1099.448358] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1100.772810] LustreError: 52935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.783249] LustreError: 52935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1101.749139] LustreError: 50231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.325861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.445467] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1103.621752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.668028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1104.686980] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1106.464968] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1108.482867] Lustre: DEBUG MARKER: Wait for df (36607) ... [ 1109.424928] Lustre: DEBUG MARKER: done [ 1114.082979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.083572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.091847] LustreError: Skipped 1 previous similar message [ 1114.115801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.231196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.234775] Lustre: Skipped 4 previous similar messages [ 1125.343289] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1125.447692] Lustre: server umount lustre-OST0000 complete [ 1130.465035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1130.466934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.470991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.475451] Lustre: Skipped 2 previous similar messages [ 1130.484456] Lustre: Skipped 3 previous similar messages [ 1133.679385] Lustre: server umount lustre-MDT0000 complete [ 1135.666513] LustreError: 50215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353102 with bad export cookie 17150898712774346986 [ 1135.666873] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.671020] LustreError: 50215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1135.850962] Lustre: server umount lustre-MDT0001 complete [ 1140.988587] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1142.797757] Key type lgssc unregistered [ 1142.994053] LNet: 55159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.004490] LNetError: 55159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.022811] LNet: Removed LNI 192.168.201.138@tcp [ 1143.576144] Key type .llcrypt unregistered [ 1143.580347] Key type ._llcrypt unregistered [ 1169.409240] Key type ._llcrypt registered [ 1169.410863] Key type .llcrypt registered [ 1169.484823] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1170.288599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.306904] alg: No test for adler32 (adler32-zlib) [ 1171.243398] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1171.382193] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1172.991183] Key type lgssc registered [ 1173.608698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1178.205955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1178.230988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.464945] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1179.479219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1179.594584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.023223] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1186.685838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.739959] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1186.758118] Lustre: 57345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1186.880332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1189.103949] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1193.699569] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1197.297576] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1201.342791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1201.461869] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1201.619916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.080991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1205.375331] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1209.686882] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1215.068579] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-10 5h12m59s [ 1216.029337] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1220.306492] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1221.729045] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1221.763350] LustreError: 57353:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1242.133422] Lustre: lustre-MDT0000: Client 0d1390c3-56b4-4939-a81e-bb7a89dcf7f0 (at 192.168.201.38@tcp) reconnecting [ 1261.831148] LustreError: 57353:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1263.257367] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1264.685679] Lustre: DEBUG MARKER: done [ 1268.193788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1268.199097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.211797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.706623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.724839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.673514] Lustre: server umount lustre-OST0000 complete [ 1278.948945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.958212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1278.966383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1278.984117] Lustre: Skipped 1 previous similar message [ 1286.115386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1286.120821] Lustre: Skipped 3 previous similar messages [ 1290.207555] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1290.212978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.223058] Lustre: Skipped 1 previous similar message [ 1290.354759] Lustre: server umount lustre-MDT0000 complete [ 1292.488218] LustreError: 57338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353259 with bad export cookie 17588946563184889860 [ 1292.489344] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1292.495454] LustreError: 57338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1292.738459] Lustre: server umount lustre-MDT0001 complete [ 1298.783446] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1300.416992] Key type lgssc unregistered [ 1300.632751] LNet: 61673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.642841] LNetError: 61673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.655691] LNet: Removed LNI 192.168.201.138@tcp [ 1301.171744] Key type .llcrypt unregistered [ 1301.172941] Key type ._llcrypt unregistered [ 1326.350702] Key type ._llcrypt registered [ 1326.352333] Key type .llcrypt registered [ 1326.437126] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1327.218376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.225240] alg: No test for adler32 (adler32-zlib) [ 1328.189383] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1328.343137] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1329.999144] Key type lgssc registered [ 1330.619418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.622974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.636667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.789316] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1333.806487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1333.900176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.496329] Lustre: Failing over lustre-MDT0000 [ 1334.722197] Lustre: server umount lustre-MDT0000 complete [ 1338.363984] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:15:03 (1786353303) [ 1342.241320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.503228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.941656] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1349.643805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.737232] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1349.752773] Lustre: 64446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1349.932313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1352.452447] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1357.209437] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1360.651667] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1364.835221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.946400] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1365.077125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.682188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1368.407761] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1372.257901] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1377.246463] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1381.346020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.349579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1381.352782] Lustre: Skipped 1 previous similar message [ 1383.343021] Lustre: server umount lustre-MDT0000 complete [ 1385.459594] LustreError: 65182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353352 with bad export cookie 10206181859523970122 [ 1385.462925] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.470964] LustreError: 65182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1385.622715] Lustre: server umount lustre-MDT0001 complete [ 1387.686173] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1390.204321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1394.967968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.615797] LustreError: 63371:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000457e2b20 x1873127192691328/t0(0) o250->MGC192.168.201.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1395.848783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.909320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1398.161425] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1401.313711] LustreError: 66295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.315093] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.321634] LustreError: 66295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1401.344073] Lustre: Skipped 1 previous similar message [ 1402.336068] Lustre: 63375:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786353353/real 1786353353] req@00000000f69d5f0d x1873127192690944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786353369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1402.401042] LustreError: 66294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.403623] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1402.410904] LustreError: 66294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1402.674634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.857589] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1406.928069] Lustre: 63375:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786353357/real 1786353357] req@00000000219d5357 x1873127192691072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786353373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1407.967152] Lustre: 63372:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786353358/real 1786353358] req@00000000cda0c644 x1873127192691200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786353374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1407.974545] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1408.466559] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1410.980675] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1412.575157] Lustre: 63372:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786353362/real 1786353362] req@00000000fb42c0f6 x1873127192691456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786353378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1418.208699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1418.209631] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.213896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.219104] Lustre: Skipped 2 previous similar messages [ 1418.221432] Lustre: Skipped 3 previous similar messages [ 1420.025922] Lustre: server umount lustre-MDT0000 complete [ 1422.038950] LustreError: 64440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353388 with bad export cookie 10206181859523972404 [ 1422.046621] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.187071] Lustre: server umount lustre-MDT0001 complete [ 1425.245141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1430.116629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.457874] LustreError: 63371:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f4859a45 x1873127192712192/t0(0) o250->MGC192.168.201.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1431.645983] LustreError: 66295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.685481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.689303] Lustre: Skipped 1 previous similar message [ 1431.719719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1434.201179] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1437.156313] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.156662] LustreError: 66294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.179138] LustreError: 66294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1439.209960] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1439.696997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.927581] Lustre: 63372:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786353390/real 1786353390] req@0000000075932290 x1873127192711808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786353406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1442.612547] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1445.353339] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1446.969175] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1450.179542] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1455.075823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1455.080718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.093927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.097188] Lustre: Skipped 1 previous similar message [ 1459.709030] Lustre: server umount lustre-MDT0000 complete [ 1460.705220] LustreError: 70888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.711402] LustreError: 70888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1461.926321] LustreError: 64438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353428 with bad export cookie 10206181859523973783 [ 1461.929095] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.934225] LustreError: 64438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1462.189780] Lustre: server umount lustre-MDT0001 complete [ 1466.139353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1468.115313] Lustre: DEBUG MARKER: files compared the same [ 1478.652660] Lustre: server umount lustre-OST0000 complete [ 1488.667913] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1490.553292] Key type lgssc unregistered [ 1490.767451] LNet: 74166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.776156] LNetError: 74166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.786688] LNet: Removed LNI 192.168.201.138@tcp [ 1491.335471] Key type .llcrypt unregistered [ 1491.337664] Key type ._llcrypt unregistered [ 1505.495552] Key type ._llcrypt registered [ 1505.496856] Key type .llcrypt registered [ 1505.571694] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1506.166046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1506.229909] alg: No test for adler32 (adler32-zlib) [ 1507.245676] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1507.420248] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1509.047153] Key type lgssc registered [ 1509.741712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1513.824925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1513.833436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.083228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.359094] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug +malloc all [ 1521.679179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.858329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1524.418073] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug +malloc all [ 1528.889659] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1532.193969] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1536.315287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1536.542841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.624646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1540.087243] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug +malloc all [ 1543.856555] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1554.911974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1554.916652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1554.933638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1556.961694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1556.962312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1556.969705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.420786] Lustre: server umount lustre-OST0000 complete [ 1562.081720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.091094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.095971] Lustre: Skipped 2 previous similar messages [ 1567.201214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1567.208229] Lustre: Skipped 1 previous similar message [ 1567.663873] Lustre: server umount lustre-MDT0000 complete [ 1569.534218] LustreError: 78339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353536 with bad export cookie 5314694832995605538 [ 1569.536739] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.544862] LustreError: 78339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1569.738329] Lustre: server umount lustre-MDT0001 complete [ 1575.499654] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1577.240873] Key type lgssc unregistered [ 1577.436706] LNet: 79092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.445215] LNetError: 79092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.458467] LNet: Removed LNI 192.168.201.138@tcp [ 1577.927653] Key type .llcrypt unregistered [ 1577.929666] Key type ._llcrypt unregistered [ 1590.775835] Key type ._llcrypt registered [ 1590.776744] Key type .llcrypt registered [ 1590.834498] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1591.250628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1591.286093] alg: No test for adler32 (adler32-zlib) [ 1592.170322] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1592.295527] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1593.895369] Key type lgssc registered [ 1594.435290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1598.560173] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1598.571340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1614.047689] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1619.167489] LustreError: 80463:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.138@tcp: failed processing log, type 4: rc = -110 [ 1645.791163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.499770] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1652.115320] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1656.235622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1656.337966] Lustre: *** cfs_fail_loc=706, val=0*** [ 1656.551681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1656.598248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1658.647172] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1663.143772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.336761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1665.433943] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1668.988389] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1671.624713] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1673.706138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1673.718180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.733137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.742331] Lustre: Skipped 1 previous similar message [ 1678.747889] Lustre: server umount lustre-OST0000 complete [ 1683.936784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.940017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.945489] Lustre: Skipped 2 previous similar messages [ 1686.377867] Lustre: server umount lustre-MDT0000 complete [ 1688.224311] LustreError: 80470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353654 with bad export cookie 17506448944851929770 [ 1688.227677] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.230807] LustreError: 80470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1688.389133] Lustre: server umount lustre-MDT0001 complete [ 1693.080219] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1694.482974] Key type lgssc unregistered [ 1694.642834] LNet: 84076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.649158] LNetError: 84076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.656542] LNet: Removed LNI 192.168.201.138@tcp [ 1695.051936] Key type .llcrypt unregistered [ 1695.053311] Key type ._llcrypt unregistered [ 1706.727489] Key type ._llcrypt registered [ 1706.730042] Key type .llcrypt registered [ 1706.771443] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1707.143591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1707.150520] alg: No test for adler32 (adler32-zlib) [ 1708.023742] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1708.160504] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1709.767879] Key type lgssc registered [ 1710.315879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1713.997428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1714.006731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1717.394810] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1720.982915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.206286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1723.017436] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1726.432133] LustreError: 86172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.438759] LustreError: 86172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1726.905547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1727.060160] LustreError: 86198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.067088] LustreError: 86198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1727.076753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.793606] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1731.172216] LustreError: 86175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.187621] LustreError: 86175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1733.955936] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.962992] Lustre: MGS: Client 062b170a-725e-4403-8d24-aaecbd45331a (at 0@lo) reconnecting [ 1733.974827] LustreError: 86186:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1734.086105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.131518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1735.886552] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1749.475046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1749.476429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1749.478739] Lustre: Skipped 1 previous similar message [ 1749.478985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1749.481559] Lustre: Skipped 1 previous similar message [ 1751.151965] Lustre: server umount lustre-OST0000 complete [ 1754.592593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.592986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1754.599290] Lustre: Skipped 1 previous similar message [ 1754.600784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.713096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.715856] Lustre: Skipped 2 previous similar messages [ 1764.833315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.836799] Lustre: Skipped 1 previous similar message [ 1767.391243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1767.452965] Lustre: server umount lustre-MDT0000 complete [ 1769.195551] Lustre: server umount lustre-MDT0001 complete [ 1770.799531] Lustre: server umount MGS complete [ 1775.801627] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:22:20 (1786353740) [ 1789.377897] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 1793.341583] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1807.090712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.545281] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1812.840647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.928373] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1812.947558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1812.986467] Lustre: lustre-MDT0001: new disk, initializing [ 1813.014604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1813.023560] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1813.026743] Lustre: 93331:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1813.030518] LustreError: 93331:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1814.579226] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1819.366215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1819.478866] Lustre: lustre-OST0000: new disk, initializing [ 1819.481029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1819.484872] Lustre: 94099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1819.522900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.240948] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1824.801328] Lustre: 94554:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000044573421 x1873127590103680/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786353807 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1827.237781] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1830.789812] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.796418] Lustre: MGS: Client c13cf344-95f1-462c-b0c9-7d324d24d778 (at 0@lo) reconnecting [ 1830.804237] LustreError: 93317:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1830.815784] Lustre: 92601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1830.831250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1830.872187] Lustre: lustre-MDT0000: new disk, initializing [ 1830.973501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.990317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1832.031665] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1832.036242] Lustre: 95166:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000031a47ad3 x1873127590109696/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786353814 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1833.153900] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1833.900795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1833.909380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1834.160280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1853.924930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.927020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1853.929343] Lustre: Skipped 1 previous similar message [ 1853.932969] Lustre: Skipped 1 previous similar message [ 1864.162851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.167441] Lustre: Skipped 3 previous similar messages [ 1867.231112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1867.306826] Lustre: server umount lustre-OST0000 complete [ 1869.282407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.282507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1875.162654] Lustre: server umount lustre-MDT0000 complete [ 1891.295118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1891.390254] Lustre: server umount lustre-MDT0001 complete [ 1892.996221] Lustre: server umount MGS complete [ 1897.842045] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:24:22 (1786353862) [ 1905.223131] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1906.408816] Key type lgssc unregistered [ 1906.548407] LNet: 97665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.552145] LNetError: 97665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.558295] LNet: Removed LNI 192.168.201.138@tcp [ 1906.933522] Key type .llcrypt unregistered [ 1906.935188] Key type ._llcrypt unregistered [ 1915.010069] Key type ._llcrypt registered [ 1915.011576] Key type .llcrypt registered [ 1915.058082] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1915.518708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1915.542185] alg: No test for adler32 (adler32-zlib) [ 1916.448813] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1916.543385] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1918.151186] Key type lgssc registered [ 1918.656093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1921.789321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.797887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.976200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.721409] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1928.789249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.960143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1930.792993] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1934.510324] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1937.344928] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1940.203627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1940.367923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.762744] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1945.915918] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1947.559036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1962.977528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1962.980163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.986678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1964.001114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1964.001408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.004827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1965.223230] Lustre: server umount lustre-OST0000 complete [ 1967.009090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.009822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.013390] Lustre: Skipped 1 previous similar message [ 1967.017103] Lustre: Skipped 2 previous similar messages [ 1972.193621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.196140] Lustre: Skipped 1 previous similar message [ 1973.047375] Lustre: server umount lustre-MDT0000 complete [ 1974.500224] LustreError: 101362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353941 with bad export cookie 9411270711754575046 [ 1974.504783] LustreError: 101362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1974.504803] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.690425] Lustre: server umount lustre-MDT0001 complete [ 1979.039744] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1980.329250] Key type lgssc unregistered [ 1980.475713] LNet: 102417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.481938] LNetError: 102417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.491921] LNet: Removed LNI 192.168.201.138@tcp [ 1980.925059] Key type .llcrypt unregistered [ 1980.926441] Key type ._llcrypt unregistered [ 1988.862654] Key type ._llcrypt registered [ 1988.864371] Key type .llcrypt registered [ 1988.904827] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1989.304787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1989.354244] alg: No test for adler32 (adler32-zlib) [ 1990.239432] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 1990.346819] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1991.935218] Key type lgssc registered [ 1992.413257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1995.139023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1995.142232] LustreError: 103479:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1995.147924] LustreError: 103480:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1995.151299] LustreError: 103479:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1995.156682] LustreError: 103479:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1995.166394] LustreError: 103479:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1995.172849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.381937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.481024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.595087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2001.351475] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2003.443969] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing lsmod [ 2004.624700] LustreError: 104680:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2004.636283] LustreError: 104684:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2004.639544] LustreError: 104680:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 2004.642653] LustreError: 104680:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2004.647544] LustreError: 104680:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2004.651526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.850496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.084443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.085966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.086228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2010.092986] Lustre: Skipped 2 previous similar messages [ 2010.106652] Lustre: Skipped 2 previous similar messages [ 2011.107954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.227167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.231980] Lustre: Skipped 4 previous similar messages [ 2020.319118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2020.402835] Lustre: server umount lustre-MDT0000 complete [ 2021.345525] LustreError: 103518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.354561] LustreError: 103518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2022.230127] LustreError: 103497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786353988 with bad export cookie 3118657975555051991 [ 2022.237815] LustreError: 103497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.242636] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.391254] LustreError: 103948:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004367d7c9 x1873127886833792/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2023.449860] Lustre: server umount lustre-MDT0001 complete [ 2034.728577] Lustre: server umount lustre-OST0000 complete [ 2038.349941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.546887] LustreError: 105750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.572489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.120287] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2043.629025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.789839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2045.400275] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2048.341376] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.793136] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2054.164867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2054.330120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.816701] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2060.463109] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2063.532330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2074.082159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2074.082632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.085423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2074.089969] Lustre: Skipped 1 previous similar message [ 2074.094271] Lustre: Skipped 1 previous similar message [ 2079.201055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2079.204064] Lustre: Skipped 1 previous similar message [ 2079.957743] Lustre: server umount lustre-OST0000 complete [ 2084.320914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.325489] Lustre: Skipped 1 previous similar message [ 2087.515046] Lustre: server umount lustre-MDT0000 complete [ 2089.058307] LustreError: 105731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354055 with bad export cookie 3118657975555052677 [ 2089.062931] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.064168] LustreError: 105731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2089.179730] Lustre: server umount lustre-MDT0001 complete [ 2093.390495] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2094.607528] Key type lgssc unregistered [ 2094.748407] LNet: 109302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.751566] LNetError: 109302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.759799] LNet: Removed LNI 192.168.201.138@tcp [ 2095.153462] Key type .llcrypt unregistered [ 2095.155793] Key type ._llcrypt unregistered [ 2106.622520] Key type ._llcrypt registered [ 2106.623833] Key type .llcrypt registered [ 2106.677237] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2107.117517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.140888] alg: No test for adler32 (adler32-zlib) [ 2108.019632] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 2108.146061] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2109.751582] Key type lgssc registered [ 2110.233269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2113.805513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2113.815885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.007394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.694847] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2120.378322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.551410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2122.282522] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2125.355697] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2127.671791] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.344560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.493707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.800601] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2135.960778] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2139.687198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2143.634040] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2144.606236] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2146.784377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2146.788916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.794818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2149.857438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.857589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2149.858575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2151.979637] Lustre: server umount lustre-OST0000 complete [ 2154.976979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.977708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2154.981420] Lustre: Skipped 1 previous similar message [ 2154.985547] Lustre: Skipped 2 previous similar messages [ 2159.673755] Lustre: server umount lustre-MDT0000 complete [ 2160.100831] LustreError: 110695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.112452] LustreError: 110695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2161.417182] LustreError: 110721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354128 with bad export cookie 9318895737792130111 [ 2161.420328] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2161.422399] LustreError: 110721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2161.557102] Lustre: server umount lustre-MDT0001 complete [ 2166.345453] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2167.693916] Key type lgssc unregistered [ 2167.863417] LNet: 114330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.866484] LNetError: 114330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.876371] LNet: Removed LNI 192.168.201.138@tcp [ 2168.314595] Key type .llcrypt unregistered [ 2168.315980] Key type ._llcrypt unregistered [ 2176.497745] Key type ._llcrypt registered [ 2176.499116] Key type .llcrypt registered [ 2176.542941] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2177.087756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.094870] alg: No test for adler32 (adler32-zlib) [ 2178.024375] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 2178.162331] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2179.768545] Key type lgssc registered [ 2180.239741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2183.666968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2183.674155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.898329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.562486] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2190.538557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.740104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2192.607263] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2196.116774] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2198.513587] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2201.321934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2201.467638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.832592] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2206.789088] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2209.638745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2221.027051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.028492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2221.031961] Lustre: Skipped 1 previous similar message [ 2221.032568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2222.372472] Lustre: server umount lustre-OST0000 complete [ 2224.103880] Lustre: server umount lustre-MDT0000 complete [ 2225.826665] LustreError: 115512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354192 with bad export cookie 3920211085047906595 [ 2225.832208] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2225.965463] Lustre: server umount lustre-MDT0001 complete [ 2230.280480] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2231.500312] Key type lgssc unregistered [ 2231.640116] LNet: 119081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.644157] LNetError: 119081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.659640] LNet: Removed LNI 192.168.201.138@tcp [ 2232.051826] Key type .llcrypt unregistered [ 2232.053127] Key type ._llcrypt unregistered [ 2242.394742] Key type ._llcrypt registered [ 2242.396794] Key type .llcrypt registered [ 2242.435261] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2242.850600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.858319] alg: No test for adler32 (adler32-zlib) [ 2243.715482] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 2243.801907] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2245.383223] Key type lgssc registered [ 2245.779856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2248.535472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2248.542627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.696402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.124955] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2254.304669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.481448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2255.995454] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2258.808448] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2260.916764] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2263.860839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2263.993627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.312357] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2269.524889] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2275.242122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2278.354387] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2279.810196] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2279.812934] Lustre: Skipped 2 previous similar messages [ 2281.445521] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2281.448995] Lustre: Skipped 2 previous similar messages [ 2287.103222] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2287.118823] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2290.652785] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.38@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2290.656234] Lustre: Skipped 2 previous similar messages [ 2296.316977] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.38@tcp 192.168.0.[2,10]@tcp [ 2297.700718] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.38/24@tcp in log lustre-MDT0000 [ 2297.705248] Lustre: Skipped 2 previous similar messages [ 2302.460295] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.38/24@tcp [ 2302.462539] Lustre: Skipped 1 previous similar message [ 2306.022449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.022885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.023199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.034320] Lustre: Skipped 1 previous similar message [ 2310.938347] Lustre: server umount lustre-OST0000 complete [ 2316.257196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.258327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2316.259422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.259426] Lustre: Skipped 1 previous similar message [ 2316.270170] Lustre: Skipped 1 previous similar message [ 2316.286380] LustreError: Skipped 1 previous similar message [ 2318.496086] Lustre: server umount lustre-MDT0000 complete [ 2320.058152] LustreError: 120455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354286 with bad export cookie 2011170903283047176 [ 2320.061740] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2320.063709] LustreError: 120455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2320.188251] Lustre: server umount lustre-MDT0001 complete [ 2324.799579] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2326.128416] Key type lgssc unregistered [ 2326.285088] LNet: 125648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.288548] LNetError: 125648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.297559] LNet: Removed LNI 192.168.201.138@tcp [ 2326.685849] Key type .llcrypt unregistered [ 2326.687415] Key type ._llcrypt unregistered [ 2339.048515] Key type ._llcrypt registered [ 2339.049950] Key type .llcrypt registered [ 2339.098792] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2339.421892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2339.430620] alg: No test for adler32 (adler32-zlib) [ 2340.319091] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 2340.423402] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2342.015176] Key type lgssc registered [ 2342.437055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.600535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2345.606738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.827726] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2346.835348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.840837] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.38/24@tcp [ 2348.446316] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2351.608048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.738817] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2351.747929] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.38/24@tcp [ 2351.770386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2353.247456] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2356.179706] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2358.390470] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2361.341175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.520553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.776947] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2366.621579] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2371.751119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2377.186181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2377.186827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.192862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.198517] Lustre: Skipped 1 previous similar message [ 2378.457912] Lustre: server umount lustre-OST0000 complete [ 2382.058311] Lustre: server umount lustre-MDT0000 complete [ 2382.306229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.317183] LustreError: 128536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.325981] LustreError: 128536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2383.681315] LustreError: 127168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354350 with bad export cookie 15006482955085140995 [ 2383.682797] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.685432] LustreError: 127168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2383.833566] Lustre: server umount lustre-MDT0001 complete [ 2388.080781] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2389.296848] Key type lgssc unregistered [ 2389.440467] LNet: 130774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2389.443266] LNetError: 130774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2389.449577] LNet: Removed LNI 192.168.201.138@tcp [ 2389.816820] Key type .llcrypt unregistered [ 2389.818130] Key type ._llcrypt unregistered [ 2400.185762] Key type ._llcrypt registered [ 2400.187045] Key type .llcrypt registered [ 2400.238124] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 2407.112548] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2407.573709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2407.581985] alg: No test for adler32 (adler32-zlib) [ 2408.452757] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 2408.553232] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2410.135155] Key type lgssc registered [ 2410.689511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2423.749061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2423.756855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.865890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2424.883685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2424.929259] Lustre: lustre-MDT0000: new disk, initializing [ 2424.966444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.979486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2426.715928] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2431.467503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.515727] Lustre: 134144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2431.540332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2431.542588] Lustre: Skipped 1 previous similar message [ 2431.579129] Lustre: lustre-MDT0001: new disk, initializing [ 2431.624815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2431.639943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2431.644163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2433.141497] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2437.382872] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2439.429935] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2442.067325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2442.193467] Lustre: lustre-OST0000: new disk, initializing [ 2442.196070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2442.198550] Lustre: 136113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2442.237327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2444.635217] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2447.360132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2447.363899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2447.410519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2448.778800] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2450.322862] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2450.404659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2451.979318] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2452.065798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2457.571873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2457.574560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.574784] LustreError: Skipped 1 previous similar message [ 2457.580301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.584625] Lustre: Skipped 1 previous similar message [ 2462.693520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2462.698331] Lustre: Skipped 1 previous similar message [ 2467.295120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2467.365069] Lustre: server umount lustre-OST0000 complete [ 2472.928550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.928830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2472.930110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2472.935020] Lustre: Skipped 2 previous similar messages [ 2478.052445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.056613] Lustre: Skipped 2 previous similar messages [ 2483.167112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2483.169152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.184719] Lustre: Skipped 1 previous similar message [ 2483.271755] Lustre: server umount lustre-MDT0000 complete [ 2484.824989] LustreError: 134137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354451 with bad export cookie 15695300441558530143 [ 2484.826467] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.835807] LustreError: 134137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2485.002927] Lustre: server umount lustre-MDT0001 complete [ 2488.924121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.207373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.132073] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2494.434808] LustreError: 138327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.439476] LustreError: 138327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2494.640326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.788559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.493613] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2499.632902] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2501.858638] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2504.547885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2504.674326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.862213] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2509.820729] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2511.480952] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2514.629409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2516.273392] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2516.357645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2520.032129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.117469] Lustre: lustre-OST0001: new disk, initializing [ 2520.119671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2520.122735] Lustre: 141715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2522.518173] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2526.050644] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2529.283655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2529.287307] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2529.322925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2530.209298] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2531.683307] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2531.764864] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2533.812770] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b2309b4b800.ost_server_uuid 50 [ 2534.406393] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b2309b4b800.ost_server_uuid in FULL state after 0 sec [ 2539.494467] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.496193] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2539.496555] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2539.499098] Lustre: Skipped 1 previous similar message [ 2542.719192] Lustre: server umount lustre-OST0001 complete [ 2544.609655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2544.609889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.610774] LustreError: 140145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.621194] Lustre: Skipped 1 previous similar message [ 2549.729511] LustreError: 140144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.734701] LustreError: 140144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2550.228395] Lustre: server umount lustre-OST0000 complete [ 2554.850505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.855419] Lustre: Skipped 1 previous similar message [ 2559.968155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.971037] Lustre: Skipped 8 previous similar messages [ 2566.111136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2566.178584] Lustre: server umount lustre-MDT0000 complete [ 2567.618117] LustreError: 138307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354534 with bad export cookie 15695300441558531018 [ 2567.621277] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2567.622047] LustreError: 138307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2567.762470] Lustre: server umount lustre-MDT0001 complete [ 2574.815145] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2575.958733] Key type lgssc unregistered [ 2576.102048] LNet: 144746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.106165] LNetError: 144746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.116576] LNet: Removed LNI 192.168.201.138@tcp [ 2576.453760] Key type .llcrypt unregistered [ 2576.455196] Key type ._llcrypt unregistered [ 2590.521446] Key type ._llcrypt registered [ 2590.522340] Key type .llcrypt registered [ 2590.560968] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 2597.259443] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2597.622523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2597.689135] alg: No test for adler32 (adler32-zlib) [ 2598.557200] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 2598.651802] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2600.247167] Key type lgssc registered [ 2600.705715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2616.141355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2616.150019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.245501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2617.261190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2617.304362] Lustre: lustre-MDT0000: new disk, initializing [ 2617.332540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.339618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2619.200521] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2624.264273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.313389] Lustre: 148872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2624.343212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2624.345162] Lustre: Skipped 1 previous similar message [ 2624.382602] Lustre: lustre-MDT0001: new disk, initializing [ 2624.431353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2624.453881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2624.463477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2626.280354] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2630.856357] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2633.158331] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2635.879271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2635.995629] Lustre: lustre-OST0000: new disk, initializing [ 2635.997936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2636.001223] Lustre: 150835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2636.034113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.494665] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2642.436929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2642.441597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2642.501122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2643.122242] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2645.545762] Lustre: Failing over lustre-OST0000 [ 2647.522145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.524748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2647.524968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2647.527726] Lustre: Skipped 1 previous similar message [ 2647.644301] Lustre: server umount lustre-OST0000 complete [ 2661.311103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2661.440703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.450579] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2663.767868] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2664.955784] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2665.188741] Lustre: Failing over lustre-MDT0000 [ 2665.195216] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.196321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2665.199678] Lustre: Skipped 1 previous similar message [ 2665.201684] Lustre: Skipped 1 previous similar message [ 2665.298973] Lustre: server umount lustre-MDT0000 complete [ 2673.632391] LustreError: 151518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.638891] LustreError: 151518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2675.188971] LustreError: 148878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.641228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.686453] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2678.693113] Lustre: 148875:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2678.751829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2678.754581] Lustre: Skipped 4 previous similar messages [ 2678.769961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2678.772655] LustreError: Skipped 1 previous similar message [ 2678.789106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.813485] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2680.256474] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2680.309762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2683.878346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2683.884527] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2683.908819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2688.995277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.995381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2689.001400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.002123] LustreError: Skipped 1 previous similar message [ 2689.005890] Lustre: Skipped 1 previous similar message [ 2691.161685] Lustre: server umount lustre-OST0000 complete [ 2694.112590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2694.113241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.120255] Lustre: Skipped 1 previous similar message [ 2694.123261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.125475] Lustre: Skipped 1 previous similar message [ 2698.790753] Lustre: server umount lustre-MDT0000 complete [ 2699.232649] LustreError: 151518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.240179] LustreError: 151518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2700.193177] LustreError: 150834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354666 with bad export cookie 13216692528126583437 [ 2700.195449] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.197755] LustreError: 150834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2700.320544] Lustre: server umount lustre-MDT0001 complete [ 2704.449810] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2705.575290] Key type lgssc unregistered [ 2705.714502] LNet: 154485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2705.717602] LNetError: 154485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2705.727314] LNet: Removed LNI 192.168.201.138@tcp [ 2706.062838] Key type .llcrypt unregistered [ 2706.064639] Key type ._llcrypt unregistered [ 2728.271320] Key type ._llcrypt registered [ 2728.272804] Key type .llcrypt registered [ 2728.322562] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2728.717287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2728.748813] alg: No test for adler32 (adler32-zlib) [ 2729.643286] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 2729.757302] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2731.343128] Key type lgssc registered [ 2731.778396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2734.943034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2734.950454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.124710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.782512] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2742.091232] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2744.292443] LustreError: 156029:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2744.297624] LustreError: 156029:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2744.301360] LustreError: 156029:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2744.435267] Lustre: server umount lustre-MDT0000 complete [ 2750.442915] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 2754.019733] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2766.894651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.980578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2766.994917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2767.033526] Lustre: lustre-MDT0000: new disk, initializing [ 2767.060511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.067964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2768.559728] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2773.250539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.304473] Lustre: 159324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2773.330847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2773.332852] Lustre: Skipped 1 previous similar message [ 2773.370845] Lustre: lustre-MDT0001: new disk, initializing [ 2773.409416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2773.424960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2773.428247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2775.009985] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2779.114817] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2781.111508] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2783.770036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2783.891183] Lustre: lustre-OST0000: new disk, initializing [ 2783.893198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2783.896766] Lustre: 161291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2783.934568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.231451] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2790.368412] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2793.994043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2793.998694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2794.037242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2797.953258] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2798.690899] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:39:23 (1786354763) [ 2901.471833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2901.472063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.472318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.481493] Lustre: Skipped 2 previous similar messages [ 2906.591497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.593357] Lustre: Skipped 4 previous similar messages [ 2911.711409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.713338] Lustre: Skipped 2 previous similar messages [ 2912.223173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2912.293795] Lustre: server umount lustre-MDT0000 complete [ 2913.670047] LustreError: 159317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354880 with bad export cookie 5109890217405856764 [ 2913.672846] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2913.675848] LustreError: 159317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2916.831691] LustreError: 160886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.831746] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.838909] LustreError: 160886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2916.847897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2918.879544] LustreError: 162809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.843621] Lustre: server umount lustre-MDT0001 complete [ 2927.377310] Lustre: server umount lustre-OST0000 complete [ 2930.766152] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 2934.082704] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2946.931123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.024365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2947.039158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2947.079822] Lustre: lustre-MDT0000: new disk, initializing [ 2947.111494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.118709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2948.756488] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2953.163124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2953.197526] Lustre: 165925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2953.220146] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2953.221840] Lustre: Skipped 1 previous similar message [ 2953.260595] Lustre: lustre-MDT0001: new disk, initializing [ 2953.308506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2953.314447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2954.822630] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2958.645784] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2960.457277] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2962.672692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.766766] Lustre: lustre-OST0000: new disk, initializing [ 2962.768313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2962.770579] Lustre: 167891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2962.797116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.799777] Lustre: Skipped 1 previous similar message [ 2963.701960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2963.706029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2963.740492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2964.679175] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2968.310143] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2969.763983] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2969.840744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2971.214379] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2971.285222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2974.177994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.178653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.180937] LustreError: Skipped 1 previous similar message [ 2974.181520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.186517] Lustre: Skipped 1 previous similar message [ 2974.188216] Lustre: Skipped 1 previous similar message [ 2978.067994] Lustre: server umount lustre-OST0000 complete [ 2983.906251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2983.910067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.915693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.917826] Lustre: Skipped 1 previous similar message [ 2985.561550] Lustre: server umount lustre-MDT0000 complete [ 2986.811984] LustreError: 165916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786354953 with bad export cookie 5109890217406015580 [ 2986.814801] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2986.815301] LustreError: 165916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2986.931328] Lustre: server umount lustre-MDT0001 complete [ 2989.788799] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:42:34 (1786354954) [ 2995.012656] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 2998.105853] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3009.332410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.409261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3009.419199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3009.456487] Lustre: lustre-MDT0000: new disk, initializing [ 3009.476618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3009.482289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3010.797731] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3014.864181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.910691] Lustre: 172637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3015.011367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3016.322698] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3019.963894] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3021.755556] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3023.966149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.067574] Lustre: 174614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3025.465503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3025.501637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3025.902404] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3029.529593] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3035.618420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3035.618660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.621082] LustreError: Skipped 1 previous similar message [ 3035.625355] Lustre: Skipped 1 previous similar message [ 3035.626058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3035.631041] Lustre: Skipped 3 previous similar messages [ 3037.459049] Lustre: server umount lustre-OST0000 complete [ 3040.738736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3053.023207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3053.084799] Lustre: server umount lustre-MDT0000 complete [ 3054.333091] LustreError: 172631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355021 with bad export cookie 5109890217406016448 [ 3054.336439] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3054.337322] LustreError: 172631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3054.456647] Lustre: server umount lustre-MDT0001 complete [ 3057.187205] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:43:42 (1786355022) [ 3062.557678] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3065.842184] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3078.146848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.220869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3078.229666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3078.231435] Lustre: Skipped 3 previous similar messages [ 3078.265183] Lustre: lustre-MDT0000: new disk, initializing [ 3078.266653] Lustre: Skipped 2 previous similar messages [ 3078.287978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.290554] Lustre: Skipped 2 previous similar messages [ 3078.296509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3078.299380] Lustre: Skipped 2 previous similar messages [ 3079.678070] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3083.832537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.884773] Lustre: 178924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3083.997635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3085.371464] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3089.082852] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3091.022428] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3093.368047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3093.465087] Lustre: 180902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3095.589396] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3099.374340] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3100.693494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3110.881732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3110.882567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.885086] LustreError: Skipped 1 previous similar message [ 3110.890976] Lustre: Skipped 3 previous similar messages [ 3110.893632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.897021] Lustre: Skipped 7 previous similar messages [ 3112.598845] Lustre: server umount lustre-OST0000 complete [ 3128.287118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3128.348484] Lustre: server umount lustre-MDT0000 complete [ 3129.617354] LustreError: 178918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355096 with bad export cookie 5109890217406017519 [ 3129.621117] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.622140] LustreError: 178918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3129.732824] Lustre: server umount lustre-MDT0001 complete [ 3133.332366] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3134.405615] Key type lgssc unregistered [ 3134.535345] LNet: 182761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3134.537956] LNetError: 182761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3134.547314] LNet: Removed LNI 192.168.201.138@tcp [ 3134.869477] Key type .llcrypt unregistered [ 3134.870866] Key type ._llcrypt unregistered [ 3144.102379] Key type ._llcrypt registered [ 3144.103316] Key type .llcrypt registered [ 3144.139746] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3144.525722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3144.546128] alg: No test for adler32 (adler32-zlib) [ 3145.408731] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 3145.503714] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3147.087203] Key type lgssc registered [ 3147.481750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3150.203490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3150.210426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.355791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.784399] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3155.775749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.905228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3157.270082] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3159.944222] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3161.813074] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3164.099139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3164.225142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.253281] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3168.824420] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3176.550519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3191.263770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3191.266623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.271709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.290812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.291270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.291710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.302298] Lustre: Skipped 1 previous similar message [ 3197.335484] Lustre: server umount lustre-OST0000 complete [ 3202.015976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3202.018682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.024106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.761917] Lustre: server umount lustre-MDT0000 complete [ 3206.075059] LustreError: 184135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355172 with bad export cookie 5717627312745940943 [ 3206.077242] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3206.079754] LustreError: 184135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3206.196510] Lustre: server umount lustre-MDT0001 complete [ 3209.861434] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3210.930803] Key type lgssc unregistered [ 3211.050356] LNet: 187707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3211.052316] LNetError: 187707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3211.060250] LNet: Removed LNI 192.168.201.138@tcp [ 3211.378885] Key type .llcrypt unregistered [ 3211.380342] Key type ._llcrypt unregistered [ 3220.505401] Key type ._llcrypt registered [ 3220.507396] Key type .llcrypt registered [ 3220.548982] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3220.964710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3220.971742] alg: No test for adler32 (adler32-zlib) [ 3221.836561] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 3221.938607] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3223.519158] Key type lgssc registered [ 3223.944200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3226.635473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3226.641958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.797898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.297220] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3232.499781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.647041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3234.171649] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3236.961408] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3239.119427] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3241.853921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.002361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.265168] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3247.095273] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3252.269873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3257.313583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.317578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3257.319261] Lustre: Skipped 1 previous similar message [ 3257.319780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3257.322842] Lustre: Skipped 1 previous similar message [ 3258.887425] Lustre: lustre-OST0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 3262.159874] Lustre: server umount lustre-OST0000 complete [ 3264.877560] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b2303723000.ost_server_uuid 50 [ 3265.476300] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3275.594193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.594769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.597909] Lustre: Skipped 1 previous similar message [ 3277.664917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3277.668117] Lustre: Skipped 2 previous similar messages [ 3281.750182] Lustre: server umount lustre-MDT0000 complete [ 3282.911990] LustreError: 189100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.917941] LustreError: 189100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3283.127234] LustreError: 189081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355249 with bad export cookie 18313380383782234759 [ 3283.131030] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3283.132885] LustreError: 189081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3283.257813] Lustre: server umount lustre-MDT0001 complete [ 3286.390909] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:47:31 (1786355251) [ 3288.985961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.158855] LustreError: 192994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.184719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3290.861263] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3294.199091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.341288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3295.838044] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3298.635743] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3300.664642] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3303.080707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3303.208894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.362595] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3308.145920] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3310.748417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3310.826812] Lustre: lustre-OST0001: new disk, initializing [ 3310.828437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3310.831818] Lustre: 195834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3311.430983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3311.435414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3311.475137] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3313.086457] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3313.515123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3317.158951] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3318.754930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.755289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3318.755909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.760037] Lustre: Skipped 1 previous similar message [ 3324.163112] Lustre: server umount lustre-OST0000 complete [ 3326.353503] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3326.435148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3327.479058] LustreError: 195922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.962785] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3328.057465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3332.596422] LustreError: 194814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.602085] LustreError: 194814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3339.232420] LustreError: 194812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.233992] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.236019] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3339.236025] Lustre: Skipped 4 previous similar messages [ 3339.239386] LustreError: 194812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3339.244012] Lustre: Skipped 1 previous similar message [ 3344.213718] Lustre: server umount lustre-OST0001 complete [ 3345.887889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3345.890140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.639942] Lustre: server umount lustre-MDT0000 complete [ 3352.943133] LustreError: 192976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355319 with bad export cookie 18313380383782235816 [ 3352.946996] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.947756] LustreError: 192976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3353.077530] Lustre: server umount lustre-MDT0001 complete [ 3361.745949] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:48:46 (1786355326) [ 3364.014517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.133374] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3364.151991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3364.232662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.235040] Lustre: Skipped 1 previous similar message [ 3365.667307] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3368.691681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.741114] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3368.757960] Lustre: 199246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3370.324886] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3372.854373] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3374.974959] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3377.678039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.743838] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3377.915425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3380.028554] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3382.786780] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3385.239397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3385.267325] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3387.561740] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3390.482996] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3393.504265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3393.505091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.508617] LustreError: Skipped 1 previous similar message [ 3393.510592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3393.519190] Lustre: Skipped 2 previous similar messages [ 3393.530594] Lustre: Skipped 5 previous similar messages [ 3397.692774] Lustre: server umount lustre-OST0000 complete [ 3400.168051] LustreError: 201097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3400.173058] LustreError: 201097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3410.402029] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3410.402631] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.406760] LustreError: Skipped 1 previous similar message [ 3410.414043] Lustre: Skipped 1 previous similar message [ 3414.560352] Lustre: server umount lustre-OST0001 complete [ 3420.128536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3425.760464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.763430] Lustre: Skipped 8 previous similar messages [ 3430.367686] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3430.445100] Lustre: server umount lustre-MDT0000 complete [ 3430.880241] LustreError: 199257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3430.886037] LustreError: 199257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3431.858222] LustreError: 199238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355398 with bad export cookie 18313380383782237174 [ 3431.860898] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.862263] LustreError: 199238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3431.993167] Lustre: server umount lustre-MDT0001 complete [ 3440.288498] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:50:05 (1786355405) [ 3445.684077] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3449.049365] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3461.766206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.855313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3461.868322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3461.907077] Lustre: lustre-MDT0000: new disk, initializing [ 3461.931756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.934397] Lustre: Skipped 3 previous similar messages [ 3461.940963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3463.641356] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3468.150415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3468.197954] Lustre: 207304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3468.224293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3468.226440] Lustre: Skipped 1 previous similar message [ 3468.261657] Lustre: lustre-MDT0001: new disk, initializing [ 3468.321300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3468.325101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3469.784028] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3473.753955] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3475.716601] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3478.044126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3478.158075] Lustre: lustre-OST0000: new disk, initializing [ 3478.160061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3478.163404] Lustre: 209273:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3480.237174] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3484.061585] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3485.546913] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3486.714802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3486.719238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3486.744688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3487.668367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3489.113744] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3489.197942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3491.809305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3491.810345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.811812] LustreError: Skipped 1 previous similar message [ 3491.814985] Lustre: Skipped 2 previous similar messages [ 3491.815577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3491.820543] Lustre: Skipped 1 previous similar message [ 3495.954150] Lustre: server umount lustre-OST0000 complete [ 3503.384900] Lustre: server umount lustre-MDT0000 complete [ 3504.789025] LustreError: 207298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355471 with bad export cookie 18313380383782238693 [ 3504.789277] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3504.795226] LustreError: 207298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3504.918228] Lustre: server umount lustre-MDT0001 complete [ 3508.212476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.412469] LustreError: 211500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.068399] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3513.218108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.780942] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3517.285636] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3519.330408] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3521.953112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3524.395562] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3527.124121] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3528.629376] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3534.836406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3536.262124] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3536.343486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3539.427704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3539.429874] LustreError: Skipped 1 previous similar message [ 3543.121289] Lustre: server umount lustre-OST0000 complete [ 3545.094715] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3546.564142] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3570.051425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.156030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.159054] Lustre: Skipped 5 previous similar messages [ 3571.999255] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3574.252538] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3574.746242] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3581.782648] Lustre: server umount lustre-OST0000 complete [ 3584.992745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3584.997205] Lustre: Skipped 5 previous similar messages [ 3589.277556] Lustre: server umount lustre-MDT0000 complete [ 3590.112167] LustreError: 211500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.117632] LustreError: 211500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3590.653305] LustreError: 211482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355557 with bad export cookie 18313380383782239575 [ 3590.656110] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3590.658447] LustreError: 211482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3590.764857] Lustre: server umount lustre-MDT0001 complete [ 3593.624348] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:52:38 (1786355558) [ 3595.788388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.320935] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3600.252108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.825091] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3604.200524] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3606.062528] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3608.426767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3610.508356] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3613.187976] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3614.664883] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3622.882325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3624.228088] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3624.296882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3626.428794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3626.499285] Lustre: lustre-OST0001: new disk, initializing [ 3626.501526] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3626.503600] Lustre: 220925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3628.406912] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3632.004154] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3633.332903] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3636.737081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3636.741044] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3636.772545] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3637.467780] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3638.706524] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3638.773741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3641.826846] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3641.827137] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3641.829168] LustreError: Skipped 3 previous similar messages [ 3641.833040] Lustre: Skipped 10 previous similar messages [ 3645.434703] Lustre: server umount lustre-OST0001 complete [ 3647.438825] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3647.512188] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3648.763897] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3648.822173] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3672.364040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3674.303344] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3676.618103] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3677.123655] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3677.667521] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3677.670564] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3677.670584] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3679.804614] Lustre: server umount lustre-OST0001 complete [ 3706.335111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3706.383358] Lustre: server umount lustre-OST0000 complete [ 3713.815722] Lustre: server umount lustre-MDT0000 complete [ 3715.048962] LustreError: 217480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355681 with bad export cookie 18313380383782240506 [ 3715.052335] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.053082] LustreError: 217480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3729.375110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3729.455837] Lustre: server umount lustre-MDT0001 complete [ 3736.736275] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:55:01 (1786355701) [ 3738.838084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.922643] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3738.932771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3739.001304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3739.003723] Lustre: Skipped 5 previous similar messages [ 3740.246256] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3743.003828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.052656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3743.064332] Lustre: 226186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3744.503648] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3746.785174] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3748.523493] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3750.714952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3750.785914] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3752.748071] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3755.053769] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3758.660677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3760.603710] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3762.874618] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3766.437791] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3767.825887] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3767.900643] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3769.328381] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3769.396538] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3771.096919] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b23054e0800.ost_server_uuid 50 [ 3771.537429] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b23054e0800.ost_server_uuid in FULL state after 0 sec [ 3771.831540] Lustre: Permanently deactivating lustre-OST0001 [ 3771.836907] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3772.546990] Lustre: Permanently reactivating lustre-OST0001 [ 3775.457142] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3775.457185] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.459370] LustreError: Skipped 5 previous similar messages [ 3775.462208] Lustre: Skipped 9 previous similar messages [ 3779.515120] Lustre: server umount lustre-OST0001 complete [ 3780.579151] LustreError: 230414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3780.583259] LustreError: 230414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 3786.836297] Lustre: server umount lustre-OST0000 complete [ 3794.278792] Lustre: server umount lustre-MDT0000 complete [ 3795.498383] LustreError: 226176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355762 with bad export cookie 18313380383782241780 [ 3795.501831] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3795.503021] LustreError: 226176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3795.612920] Lustre: server umount lustre-MDT0001 complete [ 3803.201076] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:56:08 (1786355768) [ 3805.686648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.785605] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3805.796153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3805.798638] Lustre: Skipped 5 previous similar messages [ 3807.230625] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3809.947950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.986485] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3809.989303] Lustre: Skipped 1 previous similar message [ 3810.000720] Lustre: 232868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3811.361685] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3813.763288] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3815.672952] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3817.885519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3817.961127] Lustre: Permanently deactivating lustre-OST0000 [ 3817.965358] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3818.148049] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3818.152455] LustreError: 234768:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3818.155990] LustreError: 234768:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3819.942567] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3822.336170] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3824.558469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3826.010181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3826.568591] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3828.852765] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3829.369530] Lustre: Permanently reactivating lustre-OST0000 [ 3835.895337] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3835.898374] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.901555] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3835.903988] Lustre: Skipped 1 previous similar message [ 3835.913227] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3835.920159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3847.613685] Lustre: server umount lustre-OST0001 complete [ 3854.931833] Lustre: server umount lustre-OST0000 complete [ 3862.486562] Lustre: server umount lustre-MDT0000 complete [ 3863.698084] LustreError: 232858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355830 with bad export cookie 18313380383782243418 [ 3863.700834] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.701850] LustreError: 232858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3863.818318] Lustre: server umount lustre-MDT0001 complete [ 3867.343939] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3868.325511] Key type lgssc unregistered [ 3868.447306] LNet: 237740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3868.449216] LNetError: 237740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3868.459285] LNet: Removed LNI 192.168.201.138@tcp [ 3868.727500] Key type .llcrypt unregistered [ 3868.728352] Key type ._llcrypt unregistered [ 3877.018110] Key type ._llcrypt registered [ 3877.019362] Key type .llcrypt registered [ 3877.054061] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3877.370295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3877.414076] alg: No test for adler32 (adler32-zlib) [ 3878.255185] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 3878.336470] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3879.911230] Key type lgssc registered [ 3880.243815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3883.355544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3883.365521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.500348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3885.786602] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3888.487402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.525907] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3888.536439] Lustre: 239175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3888.540529] Lustre: Permanently deactivating lustre-MDT0001 [ 3888.546577] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3888.660647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.884878] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3892.097111] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3893.225667] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3893.227754] LustreError: 239218:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3893.231271] LustreError: 239218:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3893.234117] LustreError: 239218:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3893.794759] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3895.882404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3895.991403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.864168] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3899.043289] LustreError: 241026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.045573] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 3899.048829] LustreError: 241026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3900.112534] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3902.352820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3902.424416] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3904.355163] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3906.856242] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3907.556901] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3909.607449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3909.607871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3912.646940] Lustre: Permanently reactivating lustre-MDT0001 [ 3912.652547] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3912.654996] Lustre: Skipped 2 previous similar messages [ 3915.770413] Lustre: lustre-MDT0001: Client 620bacb2-f9b7-4a16-9c73-9f68d72c8eed (at 192.168.201.38@tcp) reconnecting [ 3917.481029] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3917.982993] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3919.635346] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3919.707630] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3921.056486] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3921.127337] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3921.884158] Lustre: Permanently deactivating lustre-MDT0001 [ 3921.889349] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3921.891970] Lustre: Skipped 2 previous similar messages [ 3929.594913] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3938.274318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.276161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3938.278018] Lustre: Skipped 1 previous similar message [ 3938.279569] Lustre: Skipped 2 previous similar messages [ 3941.079940] Lustre: server umount lustre-MDT0000 complete [ 3942.253936] LustreError: 242035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786355909 with bad export cookie 18313938916185964643 [ 3942.256844] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.258291] LustreError: 242035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3942.376692] Lustre: server umount lustre-MDT0001 complete [ 3949.623691] Lustre: server umount lustre-OST0000 complete [ 3953.102059] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:58:38 (1786355918) [ 3956.419543] Lustre: server umount lustre-OST0001 complete [ 3958.399060] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 3961.351489] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3972.266901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.344406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3972.347423] Lustre: Skipped 2 previous similar messages [ 3972.354749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3972.391579] Lustre: lustre-MDT0000: new disk, initializing [ 3972.411178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.416693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3973.707413] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3977.794330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.840215] Lustre: 247526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3977.862028] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3977.863500] Lustre: Skipped 1 previous similar message [ 3977.896643] Lustre: lustre-MDT0001: new disk, initializing [ 3977.937841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3977.941728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3979.330436] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3982.943550] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3984.745133] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3987.062229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.162042] Lustre: lustre-OST0000: new disk, initializing [ 3987.163818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3987.170497] Lustre: 249496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3987.216524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.218688] Lustre: Skipped 1 previous similar message [ 3988.665060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3988.670074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3988.714091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3989.193522] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3992.887532] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3993.542504] LustreError: 249740:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3997.857418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3997.942162] Lustre: lustre-OST0001: new disk, initializing [ 3997.943968] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3997.946462] Lustre: 250612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3998.286134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3998.294168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3998.322105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4000.159705] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4003.631105] LustreError: 249740:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4003.638797] LustreError: 250267:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4003.977464] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4004.489473] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4013.727070] LustreError: 250267:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4015.099791] Lustre: lustre-OST0001: Not available for connect from 192.168.201.38@tcp (stopping) [ 4019.171569] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4019.173554] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.178223] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4019.182315] Lustre: Skipped 1 previous similar message [ 4020.415162] Lustre: server umount lustre-OST0001 complete [ 4024.292123] LustreError: 249480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.297073] LustreError: 249480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4029.414615] LustreError: 249479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.421418] LustreError: 249479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4034.531208] LustreError: 249749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4034.531319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4034.532180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4034.532184] Lustre: Skipped 1 previous similar message [ 4034.536112] LustreError: 249749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4034.540481] Lustre: Skipped 2 previous similar messages [ 4038.416636] Lustre: server umount lustre-OST0000 complete [ 4039.649419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4039.649461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.651293] LustreError: Skipped 1 previous similar message [ 4039.655535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.658140] Lustre: Skipped 2 previous similar messages [ 4045.788147] Lustre: server umount lustre-MDT0000 complete [ 4046.945139] LustreError: 247520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356013 with bad export cookie 18313938916185967674 [ 4046.948147] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4046.949787] LustreError: 247520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4053.105866] Lustre: server umount lustre-MDT0001 complete [ 4056.524292] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4057.533573] Key type lgssc unregistered [ 4057.657368] LNet: 252645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4057.660164] LNetError: 252645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4057.667361] LNet: Removed LNI 192.168.201.138@tcp [ 4057.995488] Key type .llcrypt unregistered [ 4057.996407] Key type ._llcrypt unregistered [ 4071.301734] Key type ._llcrypt registered [ 4071.302810] Key type .llcrypt registered [ 4071.337863] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4071.754910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4071.770472] alg: No test for adler32 (adler32-zlib) [ 4072.617813] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 4072.707979] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4074.287151] Key type lgssc registered [ 4074.652775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4077.171133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4077.177474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4078.256781] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4078.265241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4078.324462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4079.686848] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4082.374675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.409635] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4082.418975] Lustre: 254828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4082.509460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4083.841739] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4086.125563] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4087.922377] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4090.139356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4090.225308] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4090.290183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.168414] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4094.542688] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4097.547967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4143.585692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4143.586345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.587076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4143.593799] Lustre: Skipped 1 previous similar message [ 4144.662195] Lustre: server umount lustre-OST0000 complete [ 4146.024085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4150.018835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.127176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.011171] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4154.260746] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4163.364172] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4163.367674] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4163.367709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4189.152856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.153909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4189.155094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4189.155098] Lustre: Skipped 1 previous similar message [ 4189.156611] Lustre: Skipped 1 previous similar message [ 4189.159215] LustreError: Skipped 2 previous similar messages [ 4193.238654] Lustre: server umount lustre-OST0000 complete [ 4195.295986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4195.298602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.302359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.304454] Lustre: Skipped 1 previous similar message [ 4199.392146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4199.392293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.536091] Lustre: server umount lustre-MDT0000 complete [ 4201.728278] LustreError: 256684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356168 with bad export cookie 1131225978997023142 [ 4201.731128] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4201.732712] LustreError: 256684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4201.852578] Lustre: server umount lustre-MDT0001 complete [ 4205.192163] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4206.223799] Key type lgssc unregistered [ 4206.347369] LNet: 260253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4206.350793] LNetError: 260253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4206.359313] LNet: Removed LNI 192.168.201.138@tcp [ 4206.676552] Key type .llcrypt unregistered [ 4206.677515] Key type ._llcrypt unregistered [ 4215.267980] Key type ._llcrypt registered [ 4215.268790] Key type .llcrypt registered [ 4215.304021] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4215.569820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4215.579090] alg: No test for adler32 (adler32-zlib) [ 4216.420122] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 4216.502663] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4218.071126] Key type lgssc registered [ 4218.377872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4220.616334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4220.621389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.747154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.927875] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4225.572154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.706115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4227.037223] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4229.373836] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4231.183637] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4233.417749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4233.530640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4235.430299] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4237.953152] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4246.884678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4256.223875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4256.227417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.231408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4257.248893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.249123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4257.251113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4261.259618] Lustre: server umount lustre-OST0000 complete [ 4266.975781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4266.978075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.982030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4266.983923] Lustre: Skipped 1 previous similar message [ 4268.558278] Lustre: server umount lustre-MDT0000 complete [ 4269.688275] LustreError: 261628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356236 with bad export cookie 2662965216674329653 [ 4269.690985] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4269.691246] LustreError: 261628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4269.794427] Lustre: server umount lustre-MDT0001 complete [ 4273.057751] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4273.995134] Key type lgssc unregistered [ 4274.115419] LNet: 265971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4274.117744] LNetError: 265971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4274.125329] LNet: Removed LNI 192.168.201.138@tcp [ 4274.403891] Key type .llcrypt unregistered [ 4274.404771] Key type ._llcrypt unregistered [ 4281.032326] Key type ._llcrypt registered [ 4281.033908] Key type .llcrypt registered [ 4281.070993] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4281.382648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4281.405101] alg: No test for adler32 (adler32-zlib) [ 4282.250416] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 4282.337751] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4283.919188] Key type lgssc registered [ 4284.277787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4286.729233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4286.734331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.883393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.207063] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4291.984769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.108033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4293.418634] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4295.883277] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4297.578800] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4299.646104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4299.743349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.452180] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4303.781547] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4313.062103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4318.178127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.178765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4318.182246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.185504] Lustre: Skipped 1 previous similar message [ 4319.434546] Lustre: server umount lustre-OST0000 complete [ 4322.783909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4322.786308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.789812] Lustre: Skipped 1 previous similar message [ 4322.791300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.737543] Lustre: server umount lustre-MDT0000 complete [ 4327.914962] LustreError: 267150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356294 with bad export cookie 13549974279205232366 [ 4327.918136] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4327.918784] LustreError: 267150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4328.034189] Lustre: server umount lustre-MDT0001 complete [ 4331.472499] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4332.454905] Key type lgssc unregistered [ 4332.571561] LNet: 270863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4332.574064] LNetError: 270863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4332.580586] LNet: Removed LNI 192.168.201.138@tcp [ 4332.892523] Key type .llcrypt unregistered [ 4332.893629] Key type ._llcrypt unregistered [ 4341.347646] Key type ._llcrypt registered [ 4341.348794] Key type .llcrypt registered [ 4341.383696] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4341.722806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4341.784098] alg: No test for adler32 (adler32-zlib) [ 4342.635979] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 4342.734616] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4344.311170] Key type lgssc registered [ 4344.724552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4347.409423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4347.416018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.569820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.991293] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4353.127236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.267901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4354.715783] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4357.453954] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4359.338093] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4361.486518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4361.589285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.373368] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4365.610382] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4373.861653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4383.199876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4383.202033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.205790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.225185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.225738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4384.226251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4385.167484] Lustre: server umount lustre-OST0000 complete [ 4388.500300] Lustre: server umount lustre-MDT0000 complete [ 4389.344828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.345348] LustreError: 272280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.349048] Lustre: Skipped 1 previous similar message [ 4389.349335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4389.353405] LustreError: 272280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4389.668522] LustreError: 272238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356356 with bad export cookie 9924771060848801208 [ 4389.671447] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.672608] LustreError: 272238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4389.780620] Lustre: server umount lustre-MDT0001 complete [ 4393.168942] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4394.204925] Key type lgssc unregistered [ 4394.332335] LNet: 276577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4394.334932] LNetError: 276577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4394.341570] LNet: Removed LNI 192.168.201.138@tcp [ 4394.644865] Key type .llcrypt unregistered [ 4394.646473] Key type ._llcrypt unregistered [ 4401.470487] Key type ._llcrypt registered [ 4401.471331] Key type .llcrypt registered [ 4401.508953] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4401.957371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4401.965026] alg: No test for adler32 (adler32-zlib) [ 4402.797420] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 4402.878867] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4404.455159] Key type lgssc registered [ 4404.804331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4407.441436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4407.448773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.627503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.937961] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4412.752987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.888531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4414.289234] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4416.635703] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4418.505569] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4420.925744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4421.042617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4422.943253] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4425.424187] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4433.318548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4443.619551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4443.621080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.629195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4443.631594] Lustre: Skipped 1 previous similar message [ 4446.043906] Lustre: server umount lustre-OST0000 complete [ 4448.736911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.737684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4448.737835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4448.737840] LustreError: Skipped 1 previous similar message [ 4448.742223] Lustre: Skipped 2 previous similar messages [ 4453.784516] Lustre: server umount lustre-MDT0000 complete [ 4453.855955] LustreError: 277787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.861744] LustreError: 277787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4455.188269] LustreError: 277756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356421 with bad export cookie 4277606179945476623 [ 4455.191454] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.192548] LustreError: 277756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4455.323846] Lustre: server umount lustre-MDT0001 complete [ 4459.007336] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4459.996193] Key type lgssc unregistered [ 4460.117366] LNet: 281508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4460.120395] LNetError: 281508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4460.128312] LNet: Removed LNI 192.168.201.138@tcp [ 4460.488340] Key type .llcrypt unregistered [ 4460.493359] Key type ._llcrypt unregistered [ 4464.157549] Key type ._llcrypt registered [ 4464.159049] Key type .llcrypt registered [ 4464.199726] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4469.491778] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4475.315616] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4475.615646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4475.627456] alg: No test for adler32 (adler32-zlib) [ 4476.482721] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 4476.569313] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4478.151139] Key type lgssc registered [ 4478.499167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4489.204348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4489.209311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.286820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4490.296502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4490.333907] Lustre: lustre-MDT0000: new disk, initializing [ 4490.356100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.362094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4491.629631] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4495.586393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.632847] Lustre: 285033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4495.658671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4495.661031] Lustre: Skipped 1 previous similar message [ 4495.698610] Lustre: lustre-MDT0001: new disk, initializing [ 4495.728445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.740112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4495.743460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4496.993534] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4500.967361] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4502.866786] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4505.118955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4505.218346] Lustre: lustre-OST0000: new disk, initializing [ 4505.220289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4505.223555] Lustre: 286999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4505.255851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.134453] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4510.821988] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4512.238208] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4515.325367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4515.329627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4515.364216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4516.388729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4517.722548] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4517.791393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4520.417444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4520.418858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.421378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4520.424207] Lustre: Skipped 1 previous similar message [ 4524.566594] Lustre: server umount lustre-OST0000 complete [ 4526.559923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4526.562556] LustreError: Skipped 1 previous similar message [ 4526.564067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4526.568769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4526.570413] Lustre: Skipped 1 previous similar message [ 4530.656098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.656622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.776132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.778314] Lustre: Skipped 2 previous similar messages [ 4540.383131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4540.445752] Lustre: server umount lustre-MDT0000 complete [ 4541.721824] LustreError: 285024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356508 with bad export cookie 17599341782389328811 [ 4541.722920] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.727740] LustreError: 285024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4541.853375] Lustre: server umount lustre-MDT0001 complete [ 4544.687281] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:08:29 (1786356509) [ 4546.983678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.169440] LustreError: 289440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.177412] LustreError: 289440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4547.197832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.587651] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4551.817494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.976963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4553.916486] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4558.666888] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4562.447721] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4567.229788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4567.483380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.246914] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4576.157373] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4586.979584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.980270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4587.019272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4587.029524] LustreError: Skipped 1 previous similar message [ 4587.039346] Lustre: Skipped 1 previous similar message [ 4589.110049] Lustre: server umount lustre-OST0000 complete [ 4592.102341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.113055] Lustre: Skipped 2 previous similar messages [ 4597.217719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4597.227311] Lustre: Skipped 3 previous similar messages [ 4597.782839] Lustre: server umount lustre-MDT0000 complete [ 4601.671958] LustreError: 290176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356568 with bad export cookie 17599341782389329679 [ 4601.687970] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4602.210139] Lustre: server umount lustre-MDT0001 complete [ 4612.639169] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4615.221980] Key type lgssc unregistered [ 4615.403759] LNet: 292977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4615.407748] LNetError: 292977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4616.426334] LNet: Removed LNI 192.168.201.138@tcp [ 4617.352547] Key type .llcrypt unregistered [ 4617.355902] Key type ._llcrypt unregistered [ 4636.192474] Key type ._llcrypt registered [ 4636.194193] Key type .llcrypt registered [ 4636.248609] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4644.005622] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4644.542679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4644.552531] alg: No test for adler32 (adler32-zlib) [ 4645.462986] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 4645.602414] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4647.231178] Key type lgssc registered [ 4647.900490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4672.112701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4672.134727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.311358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4673.342687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4673.399925] Lustre: lustre-MDT0000: new disk, initializing [ 4673.462246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.473479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4675.996630] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4682.457294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.535260] Lustre: 296776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4682.598989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4682.602073] Lustre: Skipped 1 previous similar message [ 4682.646245] Lustre: lustre-MDT0001: new disk, initializing [ 4682.720667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4682.756023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4682.761332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4685.130832] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4690.403262] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4693.587895] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4697.458084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.625279] Lustre: lustre-OST2710: new disk, initializing [ 4697.628980] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4697.633649] Lustre: 298742:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4697.695331] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4700.042981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4700.047717] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4700.109775] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4700.868112] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4706.468747] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4709.946914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4710.036468] Lustre: lustre-OST03e8: new disk, initializing [ 4710.039582] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4710.043437] Lustre: 299803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4710.085192] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4712.841992] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4715.552993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4715.563448] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4715.591810] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4716.355977] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4719.475330] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4719.566684] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4721.511750] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4721.606229] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4723.654697] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4723.757611] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4725.922149] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4726.087687] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4728.799893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4728.804217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.810463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4730.853771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4730.857610] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.865605] Lustre: Skipped 2 previous similar messages [ 4733.745437] Lustre: server umount lustre-MDT0000 complete [ 4735.378519] LustreError: 297581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356702 with bad export cookie 15910863145785410760 [ 4735.382589] LustreError: 297581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4735.384234] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4735.531631] Lustre: server umount lustre-MDT0001 complete [ 4746.471345] Lustre: server umount lustre-OST2710 complete [ 4758.630194] Lustre: server umount lustre-OST03e8 complete [ 4766.730808] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4770.975196] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4786.936504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.065731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4787.086900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4787.142935] Lustre: lustre-MDT0000: new disk, initializing [ 4787.186620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4787.194757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4788.939067] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4794.518390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.588386] Lustre: 305044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4794.749710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4796.829236] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4802.452751] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4805.789501] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4809.580588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4809.707751] Lustre: lustre-OST0000: new disk, initializing [ 4809.709656] Lustre: Skipped 1 previous similar message [ 4809.711279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4809.714178] Lustre: Skipped 2 previous similar messages [ 4809.717169] Lustre: 307007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4809.778180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.782142] Lustre: Skipped 1 previous similar message [ 4812.798294] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4818.316648] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4819.988154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4819.993707] Lustre: Skipped 1 previous similar message [ 4819.996163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4820.049777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4821.015901] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4821.140356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4823.736687] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4823.868910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4825.060740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4825.061392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4825.063194] LustreError: Skipped 1 previous similar message [ 4825.063820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4825.063824] Lustre: Skipped 3 previous similar messages [ 4825.069568] Lustre: Skipped 1 previous similar message [ 4827.056728] Lustre: server umount lustre-OST0000 complete [ 4830.177886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.188962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4830.191088] Lustre: Skipped 1 previous similar message [ 4830.194216] Lustre: Skipped 1 previous similar message [ 4835.073976] Lustre: server umount lustre-MDT0000 complete [ 4837.313688] LustreError: 305034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356804 with bad export cookie 15910863145785412440 [ 4837.316981] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4837.329145] LustreError: 305034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4837.547626] Lustre: server umount lustre-MDT0001 complete [ 4842.490606] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:13:27 (1786356807) [ 4843.347892] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4844.330298] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:13:29 (1786356809) [ 4846.043501] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4859.682142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.784503] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4859.794586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4859.860062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.663521] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4865.617848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.672083] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4865.692341] Lustre: 310857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4867.923849] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4885.354242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4885.515163] Lustre: Denying initial registration attempt from nid 192.168.201.138@tcp, specified as failover [ 4885.526942] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4885.533580] LustreError: 312307:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4885.537360] LustreError: 312307:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4885.540677] LustreError: 312307:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4885.580694] Lustre: server umount lustre-OST0000 complete [ 4885.582042] LustreError: 312307:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4891.104505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.106198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4891.118982] Lustre: Skipped 1 previous similar message [ 4901.344906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4901.351937] Lustre: Skipped 4 previous similar messages [ 4903.391125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4903.466383] Lustre: server umount lustre-MDT0000 complete [ 4905.286090] LustreError: 310896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786356872 with bad export cookie 15910863145785413308 [ 4905.289655] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4905.293031] LustreError: 310896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4905.470775] Lustre: server umount lustre-MDT0001 complete [ 4910.690369] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4912.161478] Key type lgssc unregistered [ 4912.358619] LNet: 313435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4912.363268] LNetError: 313435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4913.385258] LNet: Removed LNI 192.168.201.138@tcp [ 4913.837876] Key type .llcrypt unregistered [ 4913.839381] Key type ._llcrypt unregistered [ 4919.665311] Key type ._llcrypt registered [ 4919.667061] Key type .llcrypt registered [ 4919.728895] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4920.444740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4920.505625] alg: No test for adler32 (adler32-zlib) [ 4921.456852] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 4921.638245] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4923.263278] Key type lgssc registered [ 4923.746760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4954.846617] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4959.492114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4959.500188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.688133] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4960.704735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4960.796286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4964.548495] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4974.100916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.252160] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4974.325066] Lustre: 316174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4974.905599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4979.935252] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4989.854583] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4996.577294] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5017.721764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5018.211489] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5018.629508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5027.571689] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5037.528639] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5043.682817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5043.691853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.702511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5044.193059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5044.209912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5044.221345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.272101] Lustre: Skipped 1 previous similar message [ 5047.404775] Lustre: server umount lustre-OST0000 complete [ 5054.438856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.457684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5054.462065] Lustre: Skipped 1 previous similar message [ 5059.582385] LustreError: 316181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.605270] LustreError: 316181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5059.767843] Lustre: server umount lustre-MDT0000 complete [ 5064.595250] LustreError: 316167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357031 with bad export cookie 134369279728111333 [ 5064.607525] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5064.611502] LustreError: 316167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5065.534960] Lustre: server umount lustre-MDT0001 complete [ 5077.501856] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5080.374944] Key type lgssc unregistered [ 5080.655146] LNet: 319752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5080.665780] LNetError: 319752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5081.705938] LNet: Removed LNI 192.168.201.138@tcp [ 5082.507599] Key type .llcrypt unregistered [ 5082.509731] Key type ._llcrypt unregistered [ 5105.541867] Key type ._llcrypt registered [ 5105.544589] Key type .llcrypt registered [ 5105.654621] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5106.909344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5106.931704] alg: No test for adler32 (adler32-zlib) [ 5107.962493] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 5108.194849] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5109.863377] Key type lgssc registered [ 5111.074908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5118.397807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5118.412664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.749490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.971923] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5130.214783] LustreError: 321171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5130.238693] LustreError: 321171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5132.238881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.573059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5136.599753] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5143.827821] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5150.656699] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5160.107795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5160.384043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.821574] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5174.147790] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5188.116708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.118855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5188.123923] Lustre: Skipped 2 previous similar messages [ 5188.137562] Lustre: Skipped 2 previous similar messages [ 5188.343375] Lustre: server umount lustre-MDT0000 complete [ 5191.817965] LustreError: 321152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357158 with bad export cookie 6816898675369366029 [ 5191.831025] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5192.213240] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.38@tcp (stopping) [ 5192.224611] LustreError: 323669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.185380] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.192744] LustreError: 323219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.193028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5196.234308] LustreError: 323219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5196.255834] Lustre: Skipped 1 previous similar message [ 5198.351695] Lustre: server umount lustre-MDT0001 complete [ 5204.478735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5212.753991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.738909] LustreError: 320479:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c87336d5 x1873131155257984/t0(0) o250->MGC192.168.201.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5217.022335] LustreError: 323006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.048155] LustreError: 323006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5217.100445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5217.161815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5221.273287] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5222.241059] LustreError: 323007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.266102] LustreError: 323007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5226.374886] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5228.928044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.324076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5234.273139] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5234.680961] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5242.489524] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5244.241159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5250.043695] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5251.649540] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5258.209248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5258.217827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5258.237483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5258.243442] Lustre: Skipped 1 previous similar message [ 5260.277453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.291842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5262.699973] Lustre: server umount lustre-OST0000 complete [ 5270.500226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.502180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.502279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5270.509054] Lustre: Skipped 1 previous similar message [ 5270.520255] Lustre: Skipped 3 previous similar messages [ 5272.647251] Lustre: server umount lustre-MDT0000 complete [ 5275.619889] LustreError: 324785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.649633] LustreError: 324785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5276.391834] LustreError: 321151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357243 with bad export cookie 6816898675369373162 [ 5276.404803] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.421008] LustreError: 321151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5276.943367] Lustre: server umount lustre-MDT0001 complete [ 5287.478466] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5290.440912] Key type lgssc unregistered [ 5290.803843] LNet: 327463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5290.809245] LNetError: 327463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5291.885047] LNet: Removed LNI 192.168.201.138@tcp [ 5292.747992] Key type .llcrypt unregistered [ 5292.752487] Key type ._llcrypt unregistered [ 5317.966663] Key type ._llcrypt registered [ 5317.968383] Key type .llcrypt registered [ 5318.051194] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5319.418298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5319.477849] alg: No test for adler32 (adler32-zlib) [ 5320.691855] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 5320.900328] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5322.575170] Key type lgssc registered [ 5323.737106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5330.833136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5330.848398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.532757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.428529] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5342.707463] LustreError: 328894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5342.720914] LustreError: 328894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5344.898676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5345.355228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5350.245459] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5358.347775] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5363.853630] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5367.781626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.784895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.804298] Lustre: Skipped 1 previous similar message [ 5367.830250] Lustre: Skipped 1 previous similar message [ 5370.916399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5370.932487] Lustre: Skipped 1 previous similar message [ 5373.024597] Lustre: server umount lustre-MDT0000 complete [ 5375.973245] LustreError: 329881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.996269] LustreError: 329881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5376.807400] LustreError: 328877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357343 with bad export cookie 5020307808730785969 [ 5376.831977] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5377.167027] Lustre: server umount lustre-MDT0001 complete [ 5392.938347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.288035] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5393.332056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5393.557225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.169970] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5407.588787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.748531] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5407.781414] Lustre: 331481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5408.076501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5412.590914] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5420.227528] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5426.502442] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5436.081954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5436.301771] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5436.316972] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5436.327305] LustreError: 333362:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5436.339830] LustreError: 333362:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5436.347930] LustreError: 333362:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5436.465131] Lustre: server umount lustre-OST0000 complete [ 5436.467070] LustreError: 333362:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5446.780130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5447.029444] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5447.214840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.903728] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5455.060727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5462.108583] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5470.930926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5471.067650] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5471.207405] Lustre: lustre-OST0001: new disk, initializing [ 5471.212083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5471.221820] Lustre: 334908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5471.342693] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5477.004719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5477.011255] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5477.102769] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5478.179876] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5485.672432] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5492.708308] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5492.714764] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.720416] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5492.728852] Lustre: Skipped 1 previous similar message [ 5492.742790] Lustre: Skipped 1 previous similar message [ 5494.211230] Lustre: server umount lustre-OST0001 complete [ 5497.838616] LustreError: 335208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5497.849607] LustreError: 335208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5500.898718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5500.908784] LustreError: Skipped 1 previous similar message [ 5500.916137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.955459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5502.968774] LustreError: 335208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5502.989639] LustreError: 335208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5504.478354] Lustre: server umount lustre-OST0000 complete [ 5510.624679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5510.636414] LustreError: Skipped 1 previous similar message [ 5510.639633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.651959] Lustre: Skipped 1 previous similar message [ 5510.661644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5510.679445] Lustre: Skipped 3 previous similar messages [ 5514.706641] Lustre: server umount lustre-MDT0000 complete [ 5518.045629] LustreError: 331474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357484 with bad export cookie 5020307808730786522 [ 5518.052593] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5518.054959] LustreError: 331474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5518.377945] Lustre: server umount lustre-MDT0001 complete [ 5528.788766] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5531.118327] Key type lgssc unregistered [ 5531.396595] LNet: 336867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5531.405217] LNetError: 336867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5531.443310] LNet: Removed LNI 192.168.201.138@tcp [ 5532.182368] Key type .llcrypt unregistered [ 5532.189410] Key type ._llcrypt unregistered [ 5590.476597] Key type ._llcrypt registered [ 5590.478718] Key type .llcrypt registered [ 5590.574416] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5602.027446] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5603.126276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5603.145056] alg: No test for adler32 (adler32-zlib) [ 5604.129764] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 5604.339081] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5606.015177] Key type lgssc registered [ 5607.110580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5634.550170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5634.582397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.757784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5635.781729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5635.834955] Lustre: lustre-MDT0000: new disk, initializing [ 5635.887457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.896986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5639.171545] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5648.023297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.122506] Lustre: 341780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5648.172585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5648.175257] Lustre: Skipped 1 previous similar message [ 5648.237624] Lustre: lustre-MDT0001: new disk, initializing [ 5648.317383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5648.352464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5648.365934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5652.156430] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5659.910126] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5664.595409] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5669.846652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5670.036280] Lustre: lustre-OST0000: new disk, initializing [ 5670.039402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5670.047858] Lustre: 343745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5670.120993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.300241] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5675.568193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5675.576042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5675.651975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5683.560678] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5687.390896] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5687.585977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5691.884509] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5692.043127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5696.482439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5696.484855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.492949] LustreError: Skipped 1 previous similar message [ 5696.494918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5696.510524] Lustre: Skipped 1 previous similar message [ 5696.528126] Lustre: Skipped 1 previous similar message [ 5701.606464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.609970] Lustre: Skipped 1 previous similar message [ 5706.725436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.734779] Lustre: Skipped 1 previous similar message [ 5708.255126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5708.457639] Lustre: server umount lustre-OST0000 complete [ 5711.840413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5711.850050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.854177] Lustre: Skipped 1 previous similar message [ 5716.961644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5716.971137] Lustre: Skipped 1 previous similar message [ 5717.769450] Lustre: server umount lustre-MDT0000 complete [ 5720.431013] LustreError: 341772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357687 with bad export cookie 676728655336220925 [ 5720.431807] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5720.439389] LustreError: 341772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5720.676863] Lustre: server umount lustre-MDT0001 complete [ 5726.209881] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:28:10 (1786357690) [ 5731.123689] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:28:15 (1786357695) [ 5736.598286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5736.910606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.563110] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5742.049703] LustreError: 346502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.058687] LustreError: 346502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5744.633147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.875354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.786749] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5753.068394] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5757.177602] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5761.950970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5762.152759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5766.029789] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5770.334662] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5775.874738] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5776.952838] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5777.927432] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5778.960351] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5780.278672] Lustre: Failing over lustre-MDT0000 [ 5780.506413] Lustre: server umount lustre-MDT0000 complete [ 5780.966446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5780.970347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.981938] Lustre: Skipped 1 previous similar message [ 5780.985645] LustreError: 346503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.024206] LustreError: 346497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.061470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.066453] LustreError: 346502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.077674] LustreError: 346502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5790.178434] LustreError: 346498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.188153] LustreError: 346498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5795.746205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.831150] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5795.995581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5796.031377] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5797.367911] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5798.668516] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5801.448926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5801.462770] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5801.491324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5804.990342] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5805.870457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5808.429950] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5809.721121] Lustre: Failing over lustre-MDT0000 [ 5809.941790] Lustre: server umount lustre-MDT0000 complete [ 5811.680258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5811.682131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.684324] LustreError: 350326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.684345] LustreError: 350326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5811.706736] Lustre: Skipped 3 previous similar messages [ 5825.960338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.019410] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5826.209333] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5828.089779] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5829.098276] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5831.659268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5831.664224] Lustre: Skipped 2 previous similar messages [ 5831.683455] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5831.720533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5835.849497] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5836.786453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5841.888363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.890713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5841.899213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5841.911825] Lustre: Skipped 2 previous similar messages [ 5846.888600] Lustre: server umount lustre-MDT0000 complete [ 5847.008771] LustreError: 346503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.016793] LustreError: 346503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5849.559955] LustreError: 346484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357816 with bad export cookie 676728655336223634 [ 5849.562792] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5849.577349] LustreError: 346484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5849.828718] Lustre: server umount lustre-MDT0001 complete [ 5862.991527] Lustre: server umount lustre-OST0000 complete [ 5872.527150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5872.828408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.832832] Lustre: Skipped 1 previous similar message [ 5874.816109] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5879.752116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5882.764896] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5887.153361] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5890.059735] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5894.151924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5897.463994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5898.128814] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5902.499983] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5913.061608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5913.066641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.074895] Lustre: Skipped 2 previous similar messages [ 5913.077328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5916.284865] Lustre: server umount lustre-OST0000 complete [ 5924.730745] Lustre: server umount lustre-MDT0000 complete [ 5925.346654] LustreError: 352832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.360849] LustreError: 352832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5926.753285] LustreError: 352812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357893 with bad export cookie 676728655336224376 [ 5926.759345] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5927.062267] Lustre: server umount lustre-MDT0001 complete [ 5932.973410] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5934.772104] Key type lgssc unregistered [ 5934.947044] LNet: 356364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5934.951247] LNetError: 356364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5935.979813] LNet: Removed LNI 192.168.201.138@tcp [ 5936.626500] Key type .llcrypt unregistered [ 5936.627862] Key type ._llcrypt unregistered [ 5953.196249] Key type ._llcrypt registered [ 5953.198517] Key type .llcrypt registered [ 5953.264993] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5953.925024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5953.932327] alg: No test for adler32 (adler32-zlib) [ 5954.880320] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 5955.045662] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5956.655134] Key type lgssc registered [ 5957.466115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5962.345814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5962.363407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.661124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.547879] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5972.502878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.797117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5975.628171] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5980.102711] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5983.198827] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5987.047048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.219216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.273162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5991.043017] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5995.818829] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6007.778227] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.779340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6007.783445] Lustre: Skipped 2 previous similar messages [ 6007.786342] Lustre: Skipped 1 previous similar message [ 6012.482966] Lustre: server umount lustre-MDT0000 complete [ 6012.902076] LustreError: 358666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.912325] LustreError: 358666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6014.744620] LustreError: 357838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786357981 with bad export cookie 4606060075376459110 [ 6014.748098] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6014.754672] LustreError: 357838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6014.963160] Lustre: server umount lustre-MDT0001 complete [ 6027.332749] Lustre: server umount lustre-OST0000 complete [ 6035.419262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.717839] LustreError: 361565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.771934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.507713] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6041.065365] LustreError: 361566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.775455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.967493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6046.289786] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6050.272611] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6052.910586] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6056.329597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6056.516473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.999182] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6060.660931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6064.881165] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6074.338140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.341164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6074.355304] Lustre: Skipped 1 previous similar message [ 6074.355839] Lustre: Skipped 1 previous similar message [ 6079.460026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6079.464247] Lustre: Skipped 4 previous similar messages [ 6079.972595] Lustre: server umount lustre-MDT0000 complete [ 6082.288749] LustreError: 361547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358049 with bad export cookie 4606060075376460223 [ 6082.292206] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6082.296276] LustreError: 361547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6082.562817] Lustre: server umount lustre-MDT0001 complete [ 6095.248471] Lustre: server umount lustre-OST0000 complete [ 6105.023247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.343457] LustreError: 365475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.409362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.111484] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6110.690260] LustreError: 365476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.762461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6115.687640] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6120.136028] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6123.098982] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6126.853947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6131.180701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6131.279642] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6136.061406] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6141.408280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6141.414940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.427484] Lustre: Skipped 1 previous similar message [ 6141.433536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6142.443023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6144.840554] Lustre: server umount lustre-OST0000 complete [ 6147.552843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.560383] Lustre: Skipped 1 previous similar message [ 6147.561339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6147.565721] Lustre: Skipped 3 previous similar messages [ 6157.792430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6157.794639] Lustre: Skipped 3 previous similar messages [ 6161.375141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6161.494754] Lustre: server umount lustre-MDT0000 complete [ 6162.912634] LustreError: 366202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.929719] LustreError: 366202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6164.071429] LustreError: 365457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358130 with bad export cookie 4606060075376461231 [ 6164.073071] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6164.077934] LustreError: 365457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6164.319233] Lustre: server umount lustre-MDT0001 complete [ 6170.746415] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6172.282729] Key type lgssc unregistered [ 6172.487882] LNet: 369008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6172.493891] LNetError: 369008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6172.502876] LNet: Removed LNI 192.168.201.138@tcp [ 6172.979096] Key type .llcrypt unregistered [ 6172.980422] Key type ._llcrypt unregistered [ 6188.482773] Key type ._llcrypt registered [ 6188.484295] Key type .llcrypt registered [ 6188.568926] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6189.275411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6189.292950] alg: No test for adler32 (adler32-zlib) [ 6190.272394] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 6190.448685] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6192.047151] Key type lgssc registered [ 6192.734506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6197.461253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6197.471453] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.476663] LustreError: 370426:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6197.488749] LustreError: 370426:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6197.496361] LustreError: 370426:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6197.500661] LustreError: 370426:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6197.513358] LustreError: 370426:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6202.219358] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6202.222894] LustreError: 370824:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6202.235827] LustreError: 370824:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6202.240518] LustreError: 370824:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6202.244827] LustreError: 370824:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6202.252638] LustreError: 370824:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6213.841949] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6215.671993] Key type lgssc unregistered [ 6215.892356] LNet: 371829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6215.896631] LNetError: 371829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6215.916299] LNet: Removed LNI 192.168.201.138@tcp [ 6216.498872] Key type .llcrypt unregistered [ 6216.500704] Key type ._llcrypt unregistered [ 6227.601557] Key type ._llcrypt registered [ 6227.604994] Key type .llcrypt registered [ 6227.674192] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6237.219647] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6238.087994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6238.125214] alg: No test for adler32 (adler32-zlib) [ 6239.094391] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 6239.310499] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6240.943169] Key type lgssc registered [ 6241.610499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6261.094890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6261.109667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6262.270123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6262.288553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6262.336514] Lustre: lustre-MDT0000: new disk, initializing [ 6262.376817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.391266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6264.645639] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6271.352034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6271.419952] Lustre: 374960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6271.459499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6271.462085] Lustre: Skipped 1 previous similar message [ 6271.503579] Lustre: lustre-MDT0001: new disk, initializing [ 6271.607858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6271.633935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6271.641187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6273.998747] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6279.863694] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6283.243674] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6286.894364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6287.058430] Lustre: lustre-OST0000: new disk, initializing [ 6287.061358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6287.064408] Lustre: 376930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6287.107704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6290.567329] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6293.033773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6293.039632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6293.118122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6296.299906] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6299.035151] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6299.150471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6301.726351] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6301.836743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6303.206183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6303.207427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.215226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6303.217024] Lustre: Skipped 1 previous similar message [ 6303.220397] Lustre: Skipped 1 previous similar message [ 6308.332047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6308.338566] Lustre: Skipped 1 previous similar message [ 6309.039794] Lustre: server umount lustre-OST0000 complete [ 6312.416156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6312.420712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.428740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.441368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.560656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6318.563924] Lustre: Skipped 3 previous similar messages [ 6323.680364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6323.683838] Lustre: Skipped 1 previous similar message [ 6325.215190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6325.322766] Lustre: server umount lustre-MDT0000 complete [ 6327.317326] LustreError: 374955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358294 with bad export cookie 3540709085468618322 [ 6327.321346] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6327.321937] LustreError: 374955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6327.490471] Lustre: server umount lustre-MDT0001 complete [ 6331.907532] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6332.882285] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:38:17 (1786358297) [ 6336.412749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.689392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.051652] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6342.112434] LustreError: 379387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.119328] LustreError: 379387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6344.091144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6344.264875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6346.657573] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6350.799562] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6354.231415] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6358.263545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6358.501935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6361.522839] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6365.511807] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6368.866212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6368.950865] Lustre: lustre-OST0001: new disk, initializing [ 6368.953437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6368.957864] Lustre: 382221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6372.062934] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6375.503599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6375.507776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6375.580527] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6377.836837] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6385.164955] Lustre: lustre-OST0001: Not available for connect from 192.168.201.38@tcp (stopping) [ 6386.153319] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.155021] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6386.173304] LustreError: Skipped 1 previous similar message [ 6387.285393] Lustre: server umount lustre-OST0001 complete [ 6391.271238] LustreError: 381204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6391.280544] LustreError: 381204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6396.390755] LustreError: 381203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.397848] LustreError: 381203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6401.509444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.510537] LustreError: 381204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6401.515149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6401.515157] Lustre: Skipped 2 previous similar messages [ 6401.520419] Lustre: Skipped 2 previous similar messages [ 6401.534704] LustreError: 381204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6406.625174] LustreError: 381203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6406.631406] LustreError: 381203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6406.794344] Lustre: server umount lustre-OST0000 complete [ 6410.723456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6410.728192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.989552] Lustre: server umount lustre-MDT0000 complete [ 6416.723455] LustreError: 379369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358383 with bad export cookie 3540709085468619183 [ 6416.725326] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.731562] LustreError: 379369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6431.199143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6431.416407] Lustre: server umount lustre-MDT0001 complete [ 6437.397618] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6438.881994] Key type lgssc unregistered [ 6439.025843] LNet: 384207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6439.031928] LNetError: 384207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6439.043321] LNet: Removed LNI 192.168.201.138@tcp [ 6439.582430] Key type .llcrypt unregistered [ 6439.584195] Key type ._llcrypt unregistered [ 6464.939603] Key type ._llcrypt registered [ 6464.941294] Key type .llcrypt registered [ 6465.000769] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6465.476738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6465.494423] alg: No test for adler32 (adler32-zlib) [ 6466.430143] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 6466.558291] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6468.151190] Key type lgssc registered [ 6468.713966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6472.689786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6472.703426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.830502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6473.842393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6473.940102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.472932] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6481.361345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.458666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6481.485101] Lustre: 386983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6481.645724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6484.020339] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6488.069552] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6491.325195] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6493.154200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6493.167457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6493.172765] Lustre: Skipped 1 previous similar message [ 6498.272200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6498.641643] Lustre: server umount lustre-MDT0000 complete [ 6500.585245] LustreError: 386974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358467 with bad export cookie 8806921479701250202 [ 6500.592528] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6500.757771] Lustre: server umount lustre-MDT0001 complete [ 6505.231090] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:41:10 (1786358470) [ 6508.777660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.075521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6511.467282] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6514.144749] LustreError: 389492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.155099] LustreError: 389492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6516.208893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.391488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6518.422887] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6522.075519] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6524.621843] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6527.597709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6527.709331] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6527.801101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6530.653948] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6534.044886] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6538.108933] Lustre: Permanently deactivating lustre-OST0000 [ 6538.118596] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6544.374316] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6545.900312] LustreError: 392524:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6545.903676] LustreError: 392524:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6545.906510] LustreError: 392524:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6546.771310] LustreError: 392572:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6546.776232] LustreError: 392572:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6553.749989] Lustre: server umount lustre-OST0000 complete [ 6554.985872] LustreError: 392821:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6554.990268] LustreError: 392821:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6557.667163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6557.670483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.676035] Lustre: Skipped 1 previous similar message [ 6557.678716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.685028] Lustre: Skipped 1 previous similar message [ 6560.736806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6560.737639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.859638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6565.868036] Lustre: Skipped 2 previous similar messages [ 6565.942872] Lustre: server umount lustre-MDT0000 complete [ 6567.844797] LustreError: 389471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358534 with bad export cookie 8806921479701250769 [ 6567.847707] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6567.851572] LustreError: 389471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6578.219814] Lustre: server umount lustre-MDT0001 complete [ 6582.584667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.810679] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6586.164992] LustreError: 393977:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6589.887691] Lustre: 394317:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.138@tcp [ 6590.963487] Lustre: server umount MGS complete [ 6597.059021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.309045] LustreError: 395041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6597.317538] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6597.320531] Lustre: Skipped 1 previous similar message [ 6597.347059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.119787] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6602.720626] LustreError: 395042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6602.917157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.031049] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6604.867243] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6607.982052] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6610.215478] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6610.958915] Lustre: Permanently reactivating lustre-OST0000 [ 6610.964667] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6610.967482] Lustre: Skipped 2 previous similar messages [ 6632.927277] LustreError: 396711:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6637.136220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.316830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6637.320039] Lustre: Skipped 1 previous similar message [ 6640.558621] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6644.642349] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6652.899073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.905318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.914033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.917074] Lustre: Skipped 1 previous similar message [ 6653.609133] Lustre: server umount lustre-OST0000 complete [ 6657.523387] Lustre: server umount lustre-MDT0000 complete [ 6658.019250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6658.025776] Lustre: Skipped 1 previous similar message [ 6658.033359] LustreError: 395038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6658.040652] LustreError: 395038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6659.152315] LustreError: 395022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358625 with bad export cookie 8806921479701251819 [ 6659.153551] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6659.162042] LustreError: 395022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6659.297275] Lustre: server umount lustre-MDT0001 complete [ 6664.239937] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6665.573194] Key type lgssc unregistered [ 6665.726404] LNet: 399499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6665.730405] LNetError: 399499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6665.743457] LNet: Removed LNI 192.168.201.138@tcp [ 6666.136820] Key type .llcrypt unregistered [ 6666.138392] Key type ._llcrypt unregistered [ 6674.793847] Key type ._llcrypt registered [ 6674.798891] Key type .llcrypt registered [ 6674.850127] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6682.347540] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6682.816346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6682.884159] alg: No test for adler32 (adler32-zlib) [ 6683.786383] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 6683.891593] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6685.495388] Key type lgssc registered [ 6686.051404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6699.897946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6699.904699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.993808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6701.004546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6701.041391] Lustre: lustre-MDT0000: new disk, initializing [ 6701.071681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.079648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6702.572519] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6707.226456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.272064] Lustre: 402628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6707.299785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6707.302039] Lustre: Skipped 1 previous similar message [ 6707.340711] Lustre: lustre-MDT0001: new disk, initializing [ 6707.383641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6707.397085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6707.401711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6709.126932] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6713.881215] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6716.086485] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6718.749158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.882173] Lustre: lustre-OST0000: new disk, initializing [ 6718.884534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6718.887781] Lustre: 404597:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6718.923818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.209831] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6725.756582] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6727.383666] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6727.673804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6727.677369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6727.712520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6728.475354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6729.938607] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6730.022660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6732.770058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6732.770547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.772880] LustreError: Skipped 1 previous similar message [ 6732.773929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6732.776641] Lustre: Skipped 1 previous similar message [ 6736.862109] Lustre: server umount lustre-OST0000 complete [ 6738.399992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6738.402918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.408375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6738.410737] Lustre: Skipped 1 previous similar message [ 6743.008164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6743.008768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.362282] Lustre: server umount lustre-MDT0000 complete [ 6745.721058] LustreError: 402620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358712 with bad export cookie 9625816473764849079 [ 6745.724761] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6745.725325] LustreError: 402620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6745.840261] Lustre: server umount lustre-MDT0001 complete [ 6748.951054] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6749.545710] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:45:14 (1786358714) [ 6751.960311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.152647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.751671] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6757.000200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.152526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6758.571166] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6761.131184] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.102362] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6765.466034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.574936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6767.483634] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6769.978191] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6770.658177] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6781.505461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6787.040346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6787.044167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6787.049898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6787.052076] Lustre: Skipped 1 previous similar message [ 6789.083472] Lustre: server umount lustre-OST0000 complete [ 6793.185984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6793.186445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6793.187426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6793.187432] Lustre: Skipped 2 previous similar messages [ 6793.189334] LustreError: Skipped 1 previous similar message [ 6793.202164] Lustre: Skipped 2 previous similar messages [ 6796.568996] Lustre: server umount lustre-MDT0000 complete [ 6797.861516] LustreError: 407044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358764 with bad export cookie 9625816473764849954 [ 6797.863952] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6797.865279] LustreError: 407044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6797.993129] Lustre: server umount lustre-MDT0001 complete [ 6801.878201] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6802.938899] Key type lgssc unregistered [ 6803.064278] LNet: 410807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6803.066512] LNetError: 410807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6803.075286] LNet: Removed LNI 192.168.201.138@tcp [ 6803.379278] Key type .llcrypt unregistered [ 6803.380156] Key type ._llcrypt unregistered [ 6811.396785] Key type ._llcrypt registered [ 6811.397860] Key type .llcrypt registered [ 6811.437207] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6811.567211] Key type .llcrypt unregistered [ 6811.568569] Key type ._llcrypt unregistered [ 6818.616634] Key type ._llcrypt registered [ 6818.618083] Key type .llcrypt registered [ 6818.659084] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6819.015265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6819.022991] alg: No test for adler32 (adler32-zlib) [ 6819.883416] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 6819.975744] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6821.567114] Key type lgssc registered [ 6821.930594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6824.548677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6824.555285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6825.710199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6827.087052] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6830.053371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.168147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6832.231177] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6834.687740] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6835.168250] LustreError: 412997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.174882] LustreError: 412997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6836.192317] LustreError: 413015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6836.197439] LustreError: 413015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6836.198379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6836.946252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.088408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6838.492903] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6846.431829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6846.434615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.439838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.461336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.461525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6847.465504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.669046] Lustre: server umount lustre-OST0000 complete [ 6856.162543] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6856.165482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6856.171591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6856.174405] Lustre: Skipped 1 previous similar message [ 6859.036595] Lustre: server umount lustre-MDT0000 complete [ 6860.405439] LustreError: 412977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358827 with bad export cookie 5238219719018117942 [ 6860.409290] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6860.410161] LustreError: 412977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6860.547292] Lustre: server umount lustre-MDT0001 complete [ 6864.295210] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6865.409622] Key type lgssc unregistered [ 6865.532380] LNet: 416145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6865.534877] LNetError: 416145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6865.546295] LNet: Removed LNI 192.168.201.138@tcp [ 6865.878320] Key type .llcrypt unregistered [ 6865.879853] Key type ._llcrypt unregistered [ 6875.555925] Key type ._llcrypt registered [ 6875.556800] Key type .llcrypt registered [ 6875.602326] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6875.953749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6875.963610] alg: No test for adler32 (adler32-zlib) [ 6876.849216] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 6876.945351] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6878.543182] Key type lgssc registered [ 6878.950108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6881.854561] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6881.863742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6897.311403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6902.431338] LustreError: 417512:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.138@tcp: failed processing log, type 4: rc = -110 [ 6929.057912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6931.162139] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6933.616134] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6935.927902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.105573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.131245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6937.516256] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6940.477079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.623912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6942.034134] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6944.571224] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6946.491070] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6948.835913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.837377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6948.840055] Lustre: Skipped 1 previous similar message [ 6953.953857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6953.955925] Lustre: Skipped 2 previous similar messages [ 6954.894951] Lustre: server umount lustre-OST0000 complete [ 6959.073683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.074508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6959.077613] Lustre: Skipped 1 previous similar message [ 6959.081821] Lustre: Skipped 1 previous similar message [ 6962.327797] Lustre: server umount lustre-MDT0000 complete [ 6963.654921] LustreError: 419273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786358930 with bad export cookie 13750807396048882613 [ 6963.657881] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6963.659479] LustreError: 419273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6963.774589] Lustre: server umount lustre-MDT0001 complete [ 6967.309104] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6968.337817] Key type lgssc unregistered [ 6968.467379] LNet: 421082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6968.470250] LNetError: 421082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6968.478292] LNet: Removed LNI 192.168.201.138@tcp [ 6968.778154] Key type .llcrypt unregistered [ 6968.779512] Key type ._llcrypt unregistered [ 6978.120236] Key type ._llcrypt registered [ 6978.121784] Key type .llcrypt registered [ 6978.163733] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6978.484366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6978.504093] alg: No test for adler32 (adler32-zlib) [ 6979.368861] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 6979.455349] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6981.047140] Key type lgssc registered [ 6981.436269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6984.459670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6984.466663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.638816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6987.372812] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6991.779646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.003714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6995.014871] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7000.461616] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7004.962398] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7010.370214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7010.540619] LustreError: 424296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.574464] LustreError: 424296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7010.717975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7015.939365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7016.867498] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7024.681731] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7032.361243] Lustre: server umount lustre-MDT0000 complete [ 7032.808528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7032.822896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.849906] LustreError: 423204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7033.327386] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7033.350983] Lustre: Skipped 1 previous similar message [ 7035.058843] LustreError: 422458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.38@tcp arrived at 1786359001 with bad export cookie 3760278755155065812 [ 7035.083795] LustreError: 422458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7036.906763] LustreError: 425169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786359003 with bad export cookie 3760278755155064930 [ 7036.908251] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.923188] LustreError: 425169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7038.436533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7038.438282] LustreError: 423204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.446385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.470285] LustreError: 423204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7038.490447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7041.442957] LustreError: 422477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.443265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7041.456335] LustreError: 422477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7043.288646] Lustre: server umount lustre-OST0000 complete [ 7046.570677] LustreError: 422476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7046.579305] LustreError: 422476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7049.841054] Lustre: server umount lustre-MDT0001 complete [ 7057.765084] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7059.790428] Key type lgssc unregistered [ 7060.072230] LNet: 426161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7060.080764] LNetError: 426161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7060.096336] LNet: Removed LNI 192.168.201.138@tcp [ 7060.618240] Key type .llcrypt unregistered [ 7060.619781] Key type ._llcrypt unregistered [ 7077.220666] Key type ._llcrypt registered [ 7077.222090] Key type .llcrypt registered [ 7077.288736] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7077.748542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7077.767606] alg: No test for adler32 (adler32-zlib) [ 7078.654244] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 7078.787505] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 7080.399183] Key type lgssc registered [ 7081.183400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7085.611106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7085.627808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.881654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7089.289546] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7094.743017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.030138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7097.816871] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7102.610927] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7105.943562] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7109.892944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7110.130378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.173539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7113.874290] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7118.745119] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7122.912224] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7122.914932] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7122.925609] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7124.449084] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.450072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7124.457856] Lustre: Skipped 1 previous similar message [ 7126.536309] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.38@tcp (stopping) [ 7128.744461] Lustre: server umount lustre-MDT0001 complete [ 7129.570164] LustreError: 427556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.582115] LustreError: 427556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7132.640972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7132.646714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7132.652060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7134.688702] LustreError: 427574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.121389] Lustre: server umount lustre-OST0000 complete [ 7139.810608] LustreError: 427556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.455555] Lustre: server umount lustre-MDT0000 complete [ 7148.962345] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7150.687995] Key type lgssc unregistered [ 7150.882387] LNet: 431237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7150.885782] LNetError: 431237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7150.895302] LNet: Removed LNI 192.168.201.138@tcp [ 7151.457231] Key type .llcrypt unregistered [ 7151.459085] Key type ._llcrypt unregistered [ 7167.561528] Key type ._llcrypt registered [ 7167.563420] Key type .llcrypt registered [ 7167.634864] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 7176.763312] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7177.244078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7177.295607] alg: No test for adler32 (adler32-zlib) [ 7178.191063] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 7178.358697] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 7179.991164] Key type lgssc registered [ 7180.744973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7201.726108] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7211.391110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7211.408823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7212.575637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7212.594819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7212.668631] Lustre: lustre-MDT0000: new disk, initializing [ 7212.729911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.740364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7215.190890] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7223.578923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.710078] Lustre: lustre-OST0000: new disk, initializing [ 7223.712522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7223.715338] Lustre: Skipped 1 previous similar message [ 7223.718593] Lustre: 436249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7223.760167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7227.241757] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7229.459469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7229.463549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7229.476520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7233.303779] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7243.176643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.232713] Lustre: 435228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7243.276606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7243.320052] Lustre: lustre-MDT0001: new disk, initializing [ 7243.378782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7243.410806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7243.422831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7245.462731] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7251.298636] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7251.425789] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7253.679835] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7253.818558] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7260.162027] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.38@tcp (stopping) [ 7260.641233] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.645706] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7260.647287] Lustre: Skipped 1 previous similar message [ 7262.332907] Lustre: server umount lustre-MDT0001 complete [ 7270.881941] LustreError: 435236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.889893] LustreError: 435236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7275.487909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7275.491332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.496388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7275.498144] Lustre: Skipped 2 previous similar messages [ 7276.000319] LustreError: 435239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.120385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7281.120579] LustreError: 435240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.439787] Lustre: server umount lustre-OST0000 complete [ 7289.449476] Lustre: server umount lustre-MDT0000 complete [ 7295.848969] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7297.280555] Key type lgssc unregistered [ 7297.446636] LNet: 439993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7297.450537] LNetError: 439993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7297.460456] LNet: Removed LNI 192.168.201.138@tcp [ 7297.942259] Key type .llcrypt unregistered [ 7297.944216] Key type ._llcrypt unregistered [ 7316.366054] Key type ._llcrypt registered [ 7316.367118] Key type .llcrypt registered [ 7316.418183] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7316.560663] Key type .llcrypt unregistered [ 7316.563070] Key type ._llcrypt unregistered [ 7325.291181] Key type ._llcrypt registered [ 7325.294321] Key type .llcrypt registered [ 7325.380694] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7325.973985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7326.046139] alg: No test for adler32 (adler32-zlib) [ 7326.972398] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 7327.110609] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 7328.743204] Key type lgssc registered [ 7329.405612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7343.342098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7343.363636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.491170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7344.513787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7344.558464] Lustre: lustre-MDT0000: new disk, initializing [ 7344.618411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7344.636339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7346.687611] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7352.568769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7352.639199] Lustre: 443348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7352.685822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7352.688906] Lustre: Skipped 1 previous similar message [ 7352.733293] Lustre: lustre-MDT0001: new disk, initializing [ 7352.805617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7352.836487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7352.845036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7355.016535] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7361.004374] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7364.209678] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7367.644813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.784498] Lustre: lustre-OST0000: new disk, initializing [ 7367.787963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7367.790929] Lustre: 445313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7367.839764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7369.035218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7369.039758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7369.080682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7370.358609] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7374.933590] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7378.400270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7378.403652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.408394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7379.425869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7379.426315] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7379.431784] Lustre: Skipped 1 previous similar message [ 7379.437935] Lustre: Skipped 1 previous similar message [ 7383.907331] Lustre: server umount lustre-MDT0000 complete [ 7385.340140] LustreError: 443341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786359352 with bad export cookie 9815635199051989044 [ 7385.342874] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7385.345132] LustreError: 443341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7385.497179] Lustre: server umount lustre-MDT0001 complete [ 7396.429427] Lustre: server umount lustre-OST0000 complete [ 7400.735826] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:56:05 (1786359365) [ 7404.201137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7404.408422] LustreError: 447340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7404.415941] LustreError: 447340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7404.441958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7406.141713] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7409.632700] LustreError: 447341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7410.150389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.397755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7412.521491] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7416.024613] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7418.578272] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7434.599019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7434.719795] Lustre: Found index 0 for lustre-OST0000, updating log [ 7434.834448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7437.682710] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7440.840924] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7451.241904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7456.736446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7456.741094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7456.752161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7456.757635] Lustre: Skipped 2 previous similar messages [ 7459.308465] Lustre: server umount lustre-OST0000 complete [ 7472.683227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7472.688734] Lustre: Skipped 1 previous similar message [ 7472.692389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7472.694822] Lustre: Skipped 2 previous similar messages [ 7477.218286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7477.223389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.230120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7478.835551] Lustre: server umount lustre-MDT0000 complete [ 7480.488090] LustreError: 447321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786359447 with bad export cookie 9815635199051990437 [ 7480.491834] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7480.492653] LustreError: 447321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7480.635619] Lustre: server umount lustre-MDT0001 complete [ 7484.167376] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:57:29 (1786359449) [ 7500.194407] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp [ 8066.338292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.468915] LustreError: 453769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8066.491729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8067.732480] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8070.352420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.465919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8071.736800] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8074.089461] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8076.009874] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8091.425763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.508178] Lustre: Found index 0 for lustre-OST0000, updating log [ 8091.599070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.568285] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8093.671268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 8096.305557] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8098.146551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.147041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8098.157455] Lustre: Skipped 2 previous similar messages [ 8099.169128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.188524] Lustre: server umount lustre-OST0000 complete [ 8105.504429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.505570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8105.509921] Lustre: Skipped 1 previous similar message [ 8105.511810] Lustre: Skipped 1 previous similar message [ 8109.600034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.601811] Lustre: Skipped 7 previous similar messages [ 8110.617893] Lustre: server umount lustre-MDT0000 complete [ 8110.625568] LustreError: 453788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.649137] LustreError: 453770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.654708] LustreError: 453770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8111.897151] LustreError: 453750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786360078 with bad export cookie 9815635199051991508 [ 8111.900072] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8111.901465] LustreError: 453750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8112.019350] Lustre: server umount lustre-MDT0001 complete [ 8128.927173] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp [ 8129.255864] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8132.383480] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8144.612881] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:08:29 (1786360109) [ 8150.303208] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8151.356410] Key type lgssc unregistered [ 8154.591317] LNet: 460421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8154.594180] LNetError: 460421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8222.181283] LNet: Removed LNI 192.168.201.138@tcp [ 8222.459081] Key type .llcrypt unregistered [ 8222.460200] Key type ._llcrypt unregistered [ 8228.584480] Key type ._llcrypt registered [ 8228.585316] Key type .llcrypt registered [ 8228.621528] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8229.081653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8229.089142] alg: No test for adler32 (adler32-zlib) [ 8229.937928] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 8230.018629] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8231.591166] Key type lgssc registered [ 8231.935824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8233.330949] LNet: Added LNI 192.168.201.138@tcp100 [8/256/0/180] [ 8233.616297] LNet: Added LNI 192.168.201.138@tcp101 [8/256/0/180] [ 8233.900236] LNet: Added LNI 192.168.201.138@tcp102 [8/256/0/180] [ 8234.185931] LNet: Added LNI 192.168.201.138@tcp103 [8/256/0/180] [ 8234.462027] LNet: Added LNI 192.168.201.138@tcp104 [8/256/0/180] [ 8234.750855] LNet: Added LNI 192.168.201.138@tcp105 [8/256/0/180] [ 8235.021893] LNet: Added LNI 192.168.201.138@tcp106 [8/256/0/180] [ 8235.296923] LNet: Added LNI 192.168.201.138@tcp107 [8/256/0/180] [ 8235.575820] LNet: Added LNI 192.168.201.138@tcp108 [8/256/0/180] [ 8235.845772] LNet: Added LNI 192.168.201.138@tcp109 [8/256/0/180] [ 8236.122379] LNet: Added LNI 192.168.201.138@tcp110 [8/256/0/180] [ 8236.395218] LNet: Added LNI 192.168.201.138@tcp111 [8/256/0/180] [ 8236.691613] LNet: Added LNI 192.168.201.138@tcp112 [8/256/0/180] [ 8236.986318] LNet: Added LNI 192.168.201.138@tcp113 [8/256/0/180] [ 8237.290370] LNet: Added LNI 192.168.201.138@tcp114 [8/256/0/180] [ 8237.586105] LNet: Added LNI 192.168.201.138@tcp115 [8/256/0/180] [ 8237.887424] LNet: Added LNI 192.168.201.138@tcp116 [8/256/0/180] [ 8238.178807] LNet: Added LNI 192.168.201.138@tcp117 [8/256/0/180] [ 8238.461131] LNet: Added LNI 192.168.201.138@tcp118 [8/256/0/180] [ 8238.737605] LNet: Added LNI 192.168.201.138@tcp119 [8/256/0/180] [ 8239.007350] LNet: Added LNI 192.168.201.138@tcp120 [8/256/0/180] [ 8239.294610] LNet: Added LNI 192.168.201.138@tcp121 [8/256/0/180] [ 8239.570799] LNet: Added LNI 192.168.201.138@tcp122 [8/256/0/180] [ 8239.848709] LNet: Added LNI 192.168.201.138@tcp123 [8/256/0/180] [ 8240.134535] LNet: Added LNI 192.168.201.138@tcp124 [8/256/0/180] [ 8240.408710] LNet: Added LNI 192.168.201.138@tcp125 [8/256/0/180] [ 8240.683509] LNet: Added LNI 192.168.201.138@tcp126 [8/256/0/180] [ 8240.950323] LNet: Added LNI 192.168.201.138@tcp127 [8/256/0/180] [ 8241.218383] LNet: Added LNI 192.168.201.138@tcp128 [8/256/0/180] [ 8241.507371] LNet: Added LNI 192.168.201.138@tcp129 [8/256/0/180] [ 8241.789732] LNet: Added LNI 192.168.201.138@tcp130 [8/256/0/180] [ 8242.076225] LNet: Added LNI 192.168.201.138@tcp131 [8/256/0/180] [ 8242.359426] LNet: Added LNI 192.168.201.138@tcp132 [8/256/0/180] [ 8242.644596] LNet: Added LNI 192.168.201.138@tcp133 [8/256/0/180] [ 8242.927430] LNet: Added LNI 192.168.201.138@tcp134 [8/256/0/180] [ 8243.200783] LNet: Added LNI 192.168.201.138@tcp135 [8/256/0/180] [ 8245.363254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8245.368453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.411090] Lustre: 463418:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [ 8246.430523] LNetError: 463438:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [ 8246.434909] LNetError: Protocol error connecting to 192.168.201.138@tcp118 at host 192.168.201.138:988: is it running a compatible version of Lustre? [ 8246.438539] LNetError: Skipped 4 previous similar messages [ 8246.480483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8246.489324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8246.524433] Lustre: lustre-MDT0000: new disk, initializing [ 8246.548549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8246.554096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8247.455329] LNet: 13 local NIs in recovery (showing 13): 192.168.201.138@tcp122, 192.168.201.138@tcp123, 192.168.201.138@tcp118, 192.168.201.138@tcp124, 192.168.201.138@tcp125, 192.168.201.138@tcp126, 192.168.201.138@tcp127, 192.168.201.138@tcp129, 192.168.201.138@tcp128, 192.168.201.138@tcp130, 192.168.201.138@tcp131, 192.168.201.138@tcp132, 192.168.201.138@tcp133 [ 8247.457182] LNetError: 460883:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [ 8247.466345] LNetError: 460883:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [ 8247.469119] LNetError: Protocol error connecting to 192.168.201.138@tcp117 at host 192.168.201.138:988: is it running a compatible version of Lustre? [ 8247.471875] LNetError: Skipped 13 previous similar messages [ 8247.882155] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8249.505967] LNetError: 460883:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [ 8249.508565] LNetError: 460883:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 8249.511038] LNetError: Protocol error connecting to 192.168.201.138@tcp129 at host 192.168.201.138:988: is it running a compatible version of Lustre? [ 8249.514243] LNetError: Skipped 6 previous similar messages [ 8251.867817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8251.918345] Lustre: 463435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8251.943641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8251.946216] Lustre: Skipped 1 previous similar message [ 8251.982480] Lustre: lustre-MDT0001: new disk, initializing [ 8251.990077] LNetError: 463439:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [ 8251.994463] LNetError: 463439:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 8251.995485] Lustre: 464239:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [ 8251.997957] LNetError: Protocol error connecting to 192.168.201.138@tcp134 at host 192.168.201.138:988: is it running a compatible version of Lustre? [ 8252.000428] Lustre: 464239:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8252.005719] LNetError: Skipped 3 previous similar messages [ 8252.043638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8252.056642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8252.059494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8253.338606] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8256.911266] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8258.658424] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8259.745343] LNetError: 463438:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [ 8259.749340] LNetError: 463438:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 8259.751994] LNetError: Protocol error connecting to 192.168.201.138@tcp135 at host 192.168.201.138:988: is it running a compatible version of Lustre? [ 8259.755625] LNetError: Skipped 5 previous similar messages [ 8260.806160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8260.908290] Lustre: lustre-OST0000: new disk, initializing [ 8260.910224] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8260.914265] Lustre: 465409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8260.947068] Lustre: 465414:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [ 8260.950276] Lustre: 465414:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [ 8260.966979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8262.763552] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8266.287879] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8267.935852] LNetError: 460884:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [ 8267.938559] LNetError: 460884:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8267.941382] LNetError: Protocol error connecting to 192.168.201.138@tcp135 at host 192.168.201.138:988: is it running a compatible version of Lustre? [ 8267.944473] LNetError: Skipped 1 previous similar message [ 8268.432780] Lustre: server umount lustre-OST0000 complete [ 8269.297076] Lustre: 466505:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [ 8269.299575] Lustre: 466505:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8272.863733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8272.866728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.872117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8274.400224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8274.400591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8275.731225] Lustre: server umount lustre-MDT0000 complete [ 8276.830887] LustreError: 464238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786360243 with bad export cookie 18127928797330534566 [ 8276.834150] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.834652] LustreError: 464238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8276.938447] Lustre: server umount lustre-MDT0001 complete [ 8280.187461] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8281.140994] Key type lgssc unregistered [ 8281.293832] LNet: 467321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8281.296901] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8281.298785] LNet: Removed LNI 192.168.201.138@tcp [ 8282.335195] LNet: Removed LNI 192.168.201.138@tcp100 [ 8282.336639] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8282.339170] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8282.341813] LNet: Removed LNI 192.168.201.138@tcp101 [ 8283.359093] LNet: Removed LNI 192.168.201.138@tcp102 [ 8283.360469] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8283.362628] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8283.364872] LNet: Removed LNI 192.168.201.138@tcp103 [ 8284.383272] LNet: Removed LNI 192.168.201.138@tcp104 [ 8285.407106] LNet: Removed LNI 192.168.201.138@tcp105 [ 8285.408467] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8285.410868] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8285.413379] LNet: Removed LNI 192.168.201.138@tcp106 [ 8286.431086] LNet: Removed LNI 192.168.201.138@tcp107 [ 8287.455145] LNet: Removed LNI 192.168.201.138@tcp108 [ 8288.479156] LNet: Removed LNI 192.168.201.138@tcp109 [ 8289.503165] LNet: Removed LNI 192.168.201.138@tcp110 [ 8289.505542] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8289.509043] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8289.513531] LNet: Removed LNI 192.168.201.138@tcp111 [ 8290.527164] LNet: Removed LNI 192.168.201.138@tcp112 [ 8291.551179] LNet: Removed LNI 192.168.201.138@tcp113 [ 8292.575173] LNet: Removed LNI 192.168.201.138@tcp114 [ 8293.599149] LNet: Removed LNI 192.168.201.138@tcp115 [ 8294.623139] LNet: Removed LNI 192.168.201.138@tcp116 [ 8295.647124] LNet: Removed LNI 192.168.201.138@tcp117 [ 8296.671154] LNet: Removed LNI 192.168.201.138@tcp118 [ 8297.695140] LNet: Removed LNI 192.168.201.138@tcp119 [ 8297.697244] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8297.700902] LNetError: 467321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8297.704772] LNet: Removed LNI 192.168.201.138@tcp120 [ 8298.719139] LNet: Removed LNI 192.168.201.138@tcp121 [ 8299.743122] LNet: Removed LNI 192.168.201.138@tcp122 [ 8300.767159] LNet: Removed LNI 192.168.201.138@tcp123 [ 8301.791129] LNet: Removed LNI 192.168.201.138@tcp124 [ 8302.815114] LNet: Removed LNI 192.168.201.138@tcp125 [ 8303.839145] LNet: Removed LNI 192.168.201.138@tcp126 [ 8304.863112] LNet: Removed LNI 192.168.201.138@tcp127 [ 8305.887111] LNet: Removed LNI 192.168.201.138@tcp128 [ 8305.888469] LNet: Removed LNI 192.168.201.138@tcp129 [ 8306.911154] LNet: Removed LNI 192.168.201.138@tcp130 [ 8307.935130] LNet: Removed LNI 192.168.201.138@tcp131 [ 8308.959156] LNet: Removed LNI 192.168.201.138@tcp132 [ 8309.983083] LNet: Removed LNI 192.168.201.138@tcp133 [ 8311.007079] LNet: Removed LNI 192.168.201.138@tcp134 [ 8311.019513] LNet: Removed LNI 192.168.201.138@tcp135 [ 8311.296631] Key type .llcrypt unregistered [ 8311.297862] Key type ._llcrypt unregistered [ 8318.253168] Key type ._llcrypt registered [ 8318.254197] Key type .llcrypt registered [ 8318.289293] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8323.910985] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8324.214695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8324.251573] alg: No test for adler32 (adler32-zlib) [ 8325.107916] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 8325.186852] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8326.767144] Key type lgssc registered [ 8327.098155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8337.088820] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:11:42 (1786360302) [ 8342.553340] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8343.508207] Key type lgssc unregistered [ 8343.620277] LNet: 471383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8343.622155] LNetError: 471383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8343.631295] LNet: Removed LNI 192.168.201.138@tcp [ 8343.902230] Key type .llcrypt unregistered [ 8343.903182] Key type ._llcrypt unregistered [ 8350.392157] Key type ._llcrypt registered [ 8350.393287] Key type .llcrypt registered [ 8350.431845] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8350.770674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8350.803884] alg: No test for adler32 (adler32-zlib) [ 8351.666675] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 8351.773931] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8353.351193] Key type lgssc registered [ 8353.740536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8356.779422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8356.783946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8357.856553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8357.864460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8357.899989] Lustre: lustre-MDT0000: new disk, initializing [ 8357.918335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8357.923477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8359.118329] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8362.917638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8362.953749] Lustre: 472571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8362.972493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8362.974206] Lustre: Skipped 1 previous similar message [ 8363.004415] Lustre: lustre-MDT0001: new disk, initializing [ 8363.035345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8363.047052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8363.050164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8364.222360] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8367.852548] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8369.683289] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8371.914823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8372.016417] Lustre: lustre-OST0000: new disk, initializing [ 8372.018116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8372.021800] Lustre: 474540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8372.052634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8372.142857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8372.146322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8372.181816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8374.024804] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8377.747045] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8378.942757] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing lnet_if_list [ 8380.580950] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8380.594599] LNet: Added LNI 192.168.201.138@tcp10 [8/256/0/180] [ 8382.421363] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8382.432660] LNet: Added LNI 192.168.201.138@tcp11 [8/256/0/180] [ 8383.607518] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8383.615516] LNet: Added LNI 192.168.201.138@tcp12 [8/256/0/180] [ 8384.836183] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8384.848905] LNet: Added LNI 192.168.201.138@tcp13 [8/256/0/180] [ 8386.096360] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8386.104593] LNet: Added LNI 192.168.201.138@tcp14 [8/256/0/180] [ 8387.336118] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8387.344416] LNet: Added LNI 192.168.201.138@tcp15 [8/256/0/180] [ 8388.622884] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8388.632916] LNet: Added LNI 192.168.201.138@tcp16 [8/256/0/180] [ 8390.028604] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8390.039766] LNet: Added LNI 192.168.201.138@tcp17 [8/256/0/180] [ 8391.432821] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8391.441302] LNet: Added LNI 192.168.201.138@tcp18 [8/256/0/180] [ 8392.843769] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8392.852852] LNet: Added LNI 192.168.201.138@tcp19 [8/256/0/180] [ 8394.328992] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8394.339412] LNet: Added LNI 192.168.201.138@tcp20 [8/256/0/180] [ 8395.820654] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8395.829374] LNet: Added LNI 192.168.201.138@tcp21 [8/256/0/180] [ 8397.352344] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8397.363540] LNet: Added LNI 192.168.201.138@tcp22 [8/256/0/180] [ 8398.896198] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8398.904389] LNet: Added LNI 192.168.201.138@tcp23 [8/256/0/180] [ 8400.574995] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8400.586525] LNet: Added LNI 192.168.201.138@tcp24 [8/256/0/180] [ 8402.234115] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8402.244593] LNet: Added LNI 192.168.201.138@tcp25 [8/256/0/180] [ 8403.844229] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8403.855055] LNet: Added LNI 192.168.201.138@tcp26 [8/256/0/180] [ 8405.463466] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8405.472490] LNet: Added LNI 192.168.201.138@tcp27 [8/256/0/180] [ 8407.090934] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8407.100086] LNet: Added LNI 192.168.201.138@tcp28 [8/256/0/180] [ 8408.720768] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8408.729876] LNet: Added LNI 192.168.201.138@tcp29 [8/256/0/180] [ 8410.607835] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8410.619383] LNet: Added LNI 192.168.201.138@tcp30 [8/256/0/180] [ 8412.652736] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8412.665215] LNet: Added LNI 192.168.201.138@tcp31 [8/256/0/180] [ 8414.740472] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8414.753202] LNet: Added LNI 192.168.201.138@tcp32 [8/256/0/180] [ 8416.915685] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8416.930050] LNet: Added LNI 192.168.201.138@tcp33 [8/256/0/180] [ 8419.097566] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8419.112382] LNet: Added LNI 192.168.201.138@tcp34 [8/256/0/180] [ 8421.325555] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8421.336221] LNet: Added LNI 192.168.201.138@tcp35 [8/256/0/180] [ 8423.624254] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8423.636110] LNet: Added LNI 192.168.201.138@tcp36 [8/256/0/180] [ 8425.621050] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8425.632484] LNet: Added LNI 192.168.201.138@tcp37 [8/256/0/180] [ 8427.649769] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8427.661141] LNet: Added LNI 192.168.201.138@tcp38 [8/256/0/180] [ 8430.150756] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8430.162830] LNet: Added LNI 192.168.201.138@tcp39 [8/256/0/180] [ 8432.614677] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8432.625473] LNet: Added LNI 192.168.201.138@tcp40 [8/256/0/180] [ 8435.016253] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8435.025901] LNet: Added LNI 192.168.201.138@tcp41 [8/256/0/180] [ 8437.581922] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8437.591994] LNet: Added LNI 192.168.201.138@tcp42 [8/256/0/180] [ 8440.037928] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8440.052913] LNet: Added LNI 192.168.201.138@tcp43 [8/256/0/180] [ 8442.468625] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8442.477974] LNet: Added LNI 192.168.201.138@tcp44 [8/256/0/180] [ 8444.984463] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8444.997296] LNet: Added LNI 192.168.201.138@tcp45 [8/256/0/180] [ 8447.633350] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8447.644572] LNet: Added LNI 192.168.201.138@tcp46 [8/256/0/180] [ 8450.344328] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8450.356288] LNet: Added LNI 192.168.201.138@tcp47 [8/256/0/180] [ 8453.089871] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8453.103808] LNet: Added LNI 192.168.201.138@tcp48 [8/256/0/180] [ 8455.849798] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8455.859563] LNet: Added LNI 192.168.201.138@tcp49 [8/256/0/180] [ 8458.503182] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8458.513550] LNet: Added LNI 192.168.201.138@tcp50 [8/256/0/180] [ 8469.473911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8469.475338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8469.478025] Lustre: Skipped 2 previous similar messages [ 8469.480048] Lustre: Skipped 1 previous similar message [ 8472.283304] Lustre: server umount lustre-MDT0000 complete [ 8473.480213] LustreError: 473367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786360440 with bad export cookie 16164592364430526150 [ 8473.483688] LustreError: 473367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8473.484400] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8473.585603] Lustre: server umount lustre-MDT0001 complete [ 8480.332271] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8482.994324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8483.743275] LustreError: 471844:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000064d362f4 x1873134556887936/t0(0) o250->MGC192.168.201.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8483.841264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.128555] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8487.723680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8487.814127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.054274] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8489.893194] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8490.975136] Lustre: 471847:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786360441/real 1786360441] req@0000000080193dfe x1873134556887552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786360457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8491.275313] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8492.920201] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8495.071130] Lustre: 471848:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786360445/real 1786360445] req@0000000055347214 x1873134556887680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786360461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8496.095158] Lustre: 471845:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786360446/real 1786360446] req@00000000b560d992 x1873134556887808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786360462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8500.193855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.195349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.196959] Lustre: Skipped 1 previous similar message [ 8500.199283] Lustre: Skipped 1 previous similar message [ 8505.314018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8505.315550] Lustre: Skipped 1 previous similar message [ 8505.744382] Lustre: server umount lustre-OST0000 complete [ 8508.383832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8508.386519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.390077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.551878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.553624] Lustre: Skipped 3 previous similar messages [ 8521.183095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8521.243587] Lustre: server umount lustre-MDT0000 complete [ 8522.474345] LustreError: 472564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786360489 with bad export cookie 16164592364430528166 [ 8522.477555] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.480426] LustreError: 472564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8522.592752] Lustre: server umount lustre-MDT0001 complete [ 8526.248174] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8527.265799] Key type lgssc unregistered [ 8527.423496] LNet: 501501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8527.426704] LNetError: 501501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8527.429211] LNet: Removed LNI 192.168.201.138@tcp [ 8527.430360] LNet: Removed LNI 192.168.201.138@tcp10 [ 8527.431758] LNet: Removed LNI 192.168.201.138@tcp11 [ 8527.433080] LNet: Removed LNI 192.168.201.138@tcp12 [ 8527.434372] LNet: Removed LNI 192.168.201.138@tcp13 [ 8527.435786] LNet: Removed LNI 192.168.201.138@tcp14 [ 8527.437237] LNet: Removed LNI 192.168.201.138@tcp15 [ 8527.438755] LNet: Removed LNI 192.168.201.138@tcp16 [ 8527.440402] LNet: Removed LNI 192.168.201.138@tcp17 [ 8527.442019] LNet: Removed LNI 192.168.201.138@tcp18 [ 8527.443408] LNet: Removed LNI 192.168.201.138@tcp19 [ 8527.444851] LNet: Removed LNI 192.168.201.138@tcp20 [ 8527.446168] LNet: Removed LNI 192.168.201.138@tcp21 [ 8527.448045] LNet: Removed LNI 192.168.201.138@tcp22 [ 8527.449323] LNet: Removed LNI 192.168.201.138@tcp23 [ 8527.450441] LNet: Removed LNI 192.168.201.138@tcp24 [ 8527.451824] LNet: Removed LNI 192.168.201.138@tcp25 [ 8527.453149] LNet: Removed LNI 192.168.201.138@tcp26 [ 8527.454376] LNet: Removed LNI 192.168.201.138@tcp27 [ 8527.455646] LNet: Removed LNI 192.168.201.138@tcp28 [ 8527.456801] LNet: Removed LNI 192.168.201.138@tcp29 [ 8527.458057] LNet: Removed LNI 192.168.201.138@tcp30 [ 8527.459238] LNet: Removed LNI 192.168.201.138@tcp31 [ 8527.460461] LNet: Removed LNI 192.168.201.138@tcp32 [ 8527.461736] LNet: Removed LNI 192.168.201.138@tcp33 [ 8527.462951] LNet: Removed LNI 192.168.201.138@tcp34 [ 8527.464262] LNet: Removed LNI 192.168.201.138@tcp35 [ 8527.465582] LNet: Removed LNI 192.168.201.138@tcp36 [ 8527.467713] LNet: Removed LNI 192.168.201.138@tcp37 [ 8527.469754] LNet: Removed LNI 192.168.201.138@tcp38 [ 8527.471645] LNet: Removed LNI 192.168.201.138@tcp39 [ 8527.473563] LNet: Removed LNI 192.168.201.138@tcp40 [ 8527.475744] LNet: Removed LNI 192.168.201.138@tcp41 [ 8527.477942] LNet: Removed LNI 192.168.201.138@tcp42 [ 8527.480069] LNet: Removed LNI 192.168.201.138@tcp43 [ 8527.482301] LNet: Removed LNI 192.168.201.138@tcp44 [ 8527.484341] LNet: Removed LNI 192.168.201.138@tcp45 [ 8527.486415] LNet: Removed LNI 192.168.201.138@tcp46 [ 8527.488382] LNet: Removed LNI 192.168.201.138@tcp47 [ 8527.490451] LNet: Removed LNI 192.168.201.138@tcp48 [ 8527.492640] LNet: Removed LNI 192.168.201.138@tcp49 [ 8527.500370] LNet: Removed LNI 192.168.201.138@tcp50 [ 8527.811955] Key type .llcrypt unregistered [ 8527.813083] Key type ._llcrypt unregistered [ 8534.172020] Key type ._llcrypt registered [ 8534.173151] Key type .llcrypt registered [ 8534.214649] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8540.256302] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8540.574871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8540.583162] alg: No test for adler32 (adler32-zlib) [ 8541.435890] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 8541.527457] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8543.111157] Key type lgssc registered [ 8543.480302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8554.882251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8554.887543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.969645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8555.980426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8556.020437] Lustre: lustre-MDT0000: new disk, initializing [ 8556.041418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.049868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8557.370763] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8561.519360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.566253] Lustre: 504671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8561.590442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8561.592824] Lustre: Skipped 1 previous similar message [ 8561.631777] Lustre: lustre-MDT0001: new disk, initializing [ 8561.665815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8561.678818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8561.684469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8563.004022] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8566.668416] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8568.471637] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8570.936444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8571.007893] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8571.035391] Lustre: lustre-OST0000: new disk, initializing [ 8571.036835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8571.039489] Lustre: 506688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8571.068440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8572.967517] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8576.736080] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8577.533394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8577.538269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8577.581642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8582.624884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8582.626737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.628503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8582.631016] Lustre: Skipped 1 previous similar message [ 8582.634395] Lustre: Skipped 1 previous similar message [ 8585.300578] Lustre: server umount lustre-OST0000 complete [ 8587.232091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8587.234310] LustreError: Skipped 1 previous similar message [ 8587.235677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.239678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8592.727637] Lustre: server umount lustre-MDT0000 complete [ 8593.901262] LustreError: 504664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786360560 with bad export cookie 8185662088576037407 [ 8593.904084] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8593.905602] LustreError: 504664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8594.018172] Lustre: server umount lustre-MDT0001 complete [ 8597.366064] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8598.330867] Key type lgssc unregistered [ 8598.452297] LNet: 508592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8598.454147] LNetError: 508592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8598.463253] LNet: Removed LNI 192.168.201.138@tcp [ 8598.742131] Key type .llcrypt unregistered [ 8598.743067] Key type ._llcrypt unregistered [ 8605.796868] Key type ._llcrypt registered [ 8605.797736] Key type .llcrypt registered [ 8605.831520] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8606.158072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8606.188927] alg: No test for adler32 (adler32-zlib) [ 8607.042426] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 8607.130695] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8608.711151] Key type lgssc registered [ 8609.045920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8611.401675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8611.407335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8612.540385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8613.754533] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8616.356450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.464666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8617.679474] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8619.843080] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8621.505662] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8623.627643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8623.734619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8625.522958] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8627.739193] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8642.157808] loop0: detected capacity change from 0 to 204800000 [ 8642.222795] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8642.227295] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8642.232056] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8644.325571] loop0: detected capacity change from 0 to 204800000 [ 8644.436246] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8644.439766] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8644.443665] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8646.220420] loop0: detected capacity change from 0 to 204800000 [ 8648.253796] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8648.306159] Lustre: fs15246-OST0000: new disk, initializing [ 8648.307856] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8648.310503] Lustre: 513450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8648.331078] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.325341] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8654.228410] loop1: detected capacity change from 0 to 204800000 [ 8656.109447] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.146060] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8656.165948] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8656.201050] Lustre: fs15246-MDT0000: new disk, initializing [ 8656.246931] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8656.258015] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8656.295753] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8657.311882] Lustre: 514573:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000075ab9f07 x1873134824226304/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786360640 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8657.599626] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8665.573812] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.399203] LustreError: 514573:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8666.404059] LustreError: 514573:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8666.406691] LustreError: 514573:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8670.690351] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.060092] Lustre: server umount fs15246-MDT0000 complete [ 8673.911861] Lustre: server umount fs15246-OST0000 complete [ 8678.367781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8678.370298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8678.374192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8679.391725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8679.394779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8680.929111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8680.931488] Lustre: Skipped 2 previous similar messages [ 8683.284769] Lustre: server umount lustre-OST0000 complete [ 8686.048328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.051649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8686.053160] Lustre: Skipped 1 previous similar message [ 8690.518306] Lustre: server umount lustre-MDT0000 complete [ 8691.169526] LustreError: 509838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8691.175428] LustreError: 509838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8691.727416] LustreError: 514567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786360658 with bad export cookie 16454793470081166169 [ 8691.730870] LustreError: 514567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8691.731242] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8691.850276] Lustre: server umount lustre-MDT0001 complete [ 8695.612657] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8696.533914] Key type lgssc unregistered [ 8696.653321] LNet: 517666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8696.655353] LNetError: 517666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8696.668305] LNet: Removed LNI 192.168.201.138@tcp [ 8696.970535] Key type .llcrypt unregistered [ 8696.971461] Key type ._llcrypt unregistered [ 8703.950284] Key type ._llcrypt registered [ 8703.951141] Key type .llcrypt registered [ 8703.983934] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8704.359399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8704.370453] alg: No test for adler32 (adler32-zlib) [ 8705.216920] Lustre: Lustre: Build Version: 2.17.56_50_g4393166 [ 8705.299872] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8706.879132] Key type lgssc registered [ 8707.195960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8719.402425] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8722.285112] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8732.716767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8732.721673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.792881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8733.802618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8733.839569] Lustre: lustre-MDT0000: new disk, initializing [ 8733.858788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8733.864577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8735.110176] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8739.033818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8739.075708] Lustre: 522102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8739.100853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8739.103349] Lustre: Skipped 1 previous similar message [ 8739.137533] Lustre: lustre-MDT0001: new disk, initializing [ 8739.167322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8739.181548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8739.184722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8740.438968] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8743.942576] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8745.688822] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8747.783504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8747.877411] Lustre: lustre-OST0000: new disk, initializing [ 8747.880535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8747.883062] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8747.887357] Lustre: 524067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8747.916580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8749.649253] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8753.083341] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8753.130305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8753.133071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8753.144728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8754.368654] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8754.433589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8755.664621] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8755.729055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8758.241384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8758.242578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.243276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8758.245771] Lustre: Skipped 1 previous similar message [ 8758.249203] Lustre: Skipped 1 previous similar message [ 8762.454440] Lustre: server umount lustre-OST0000 complete [ 8764.895828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8764.897873] LustreError: Skipped 1 previous similar message [ 8764.899625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.903700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8768.480336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8768.480847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.757823] Lustre: server umount lustre-MDT0000 complete [ 8770.991206] LustreError: 522092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786360737 with bad export cookie 11840450878578675680 [ 8770.994373] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8770.994599] LustreError: 522092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8771.117706] Lustre: server umount lustre-MDT0001 complete [ 8779.211845] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 8782.503430] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8792.738137] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8570 sec ========== 07:19:17 (1786360757) [ 8793.275692] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:19:18 (1786360758) === [ 8794.407117] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:19:19 (1786360759) === [ 8807.289477] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8808.272848] Key type lgssc unregistered [ 8808.388303] LNet: 530700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8808.392757] LNetError: 530700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8808.400294] LNet: Removed LNI 192.168.201.138@tcp [ 8808.688859] Key type .llcrypt unregistered [ 8808.690491] Key type ._llcrypt unregistered