[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 727414247 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002390] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.007031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009231] LSM: Security Framework initializing [ 0.010080] Yama: becoming mindful. [ 0.012075] SELinux: Initializing. [ 0.013115] *** VALIDATE selinux *** [ 0.023246] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029015] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030251] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031182] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033230] *** VALIDATE tmpfs *** [ 0.035430] *** VALIDATE proc *** [ 0.037337] *** VALIDATE cgroup *** [ 0.038015] *** VALIDATE cgroup2 *** [ 0.039361] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040201] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042038] Spectre V2 : User space: Vulnerable [ 0.043016] Speculative Store Bypass: Vulnerable [ 0.046323] debug: unmapping init [mem 0xffffffffb3659000-0xffffffffb3660fff] [ 0.049000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049948] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051037] ... version: 2 [ 0.052023] ... bit width: 48 [ 0.053034] ... generic registers: 4 [ 0.054022] ... value mask: 0000ffffffffffff [ 0.055027] ... max period: 00007fffffffffff [ 0.056025] ... fixed-purpose events: 3 [ 0.057018] ... event mask: 000000070000000f [ 0.058500] rcu: Hierarchical SRCU implementation. [ 0.061057] smp: Bringing up secondary CPUs ... [ 0.062661] x86: Booting SMP configuration: [ 0.063048] .... node #0, CPUs: #1 #2 #3 [ 0.068091] smp: Brought up 1 node, 4 CPUs [ 0.070019] smpboot: Max logical packages: 1 [ 0.071026] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.103576] node 0 deferred pages initialised in 29ms [ 0.109285] devtmpfs: initialized [ 0.110396] x86/mm: Memory block size: 128MB [ 0.113921] gcov: version magic: 0x41383552 [ 0.115896] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.116099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.117408] pinctrl core: initialized pinctrl subsystem [ 0.118262] [ 0.118956] ************************************************************* [ 0.119021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.120026] ** ** [ 0.121018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.122014] ** ** [ 0.123149] ** This means that this kernel is built to expose internal ** [ 0.124015] ** IOMMU data structures, which may compromise security on ** [ 0.125019] ** your system. ** [ 0.126030] ** ** [ 0.127020] ** If you see this message and you are not debugging the ** [ 0.128019] ** kernel, report this immediately to your vendor! ** [ 0.129019] ** ** [ 0.130017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131014] ************************************************************* [ 0.133020] NET: Registered protocol family 16 [ 0.135314] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.136115] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.137089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.141304] cpuidle: using governor menu [ 0.150332] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.161079] PCI: Using configuration type 1 for base access [ 0.167168] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194393] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206000] cryptd: max_cpu_qlen set to 1000 [ 0.209624] ACPI: Added _OSI(Module Device) [ 0.211019] ACPI: Added _OSI(Processor Device) [ 0.214019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219388] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.275707] ACPI: Interpreter enabled [ 0.279567] ACPI: PM: (supports S0 S3 S4 S5) [ 0.282017] ACPI: Using IOAPIC for interrupt routing [ 0.288541] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.297129] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.340332] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.345441] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.350000] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.356871] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.370716] acpiphp: Slot [2] registered [ 0.374705] acpiphp: Slot [5] registered [ 0.379215] acpiphp: Slot [6] registered [ 0.381000] acpiphp: Slot [7] registered [ 0.383869] acpiphp: Slot [8] registered [ 0.387835] acpiphp: Slot [9] registered [ 0.392171] acpiphp: Slot [10] registered [ 0.396158] acpiphp: Slot [3] registered [ 0.398184] acpiphp: Slot [4] registered [ 0.399130] acpiphp: Slot [11] registered [ 0.402056] acpiphp: Slot [12] registered [ 0.403129] acpiphp: Slot [13] registered [ 0.407137] acpiphp: Slot [14] registered [ 0.410181] acpiphp: Slot [15] registered [ 0.412337] acpiphp: Slot [16] registered [ 0.415176] acpiphp: Slot [17] registered [ 0.418212] acpiphp: Slot [18] registered [ 0.420111] acpiphp: Slot [19] registered [ 0.422137] acpiphp: Slot [20] registered [ 0.425139] acpiphp: Slot [21] registered [ 0.427133] acpiphp: Slot [22] registered [ 0.430018] acpiphp: Slot [23] registered [ 0.432142] acpiphp: Slot [24] registered [ 0.435132] acpiphp: Slot [25] registered [ 0.436351] acpiphp: Slot [26] registered [ 0.439287] acpiphp: Slot [27] registered [ 0.441273] acpiphp: Slot [28] registered [ 0.444750] acpiphp: Slot [29] registered [ 0.446134] acpiphp: Slot [30] registered [ 0.449195] acpiphp: Slot [31] registered [ 0.452066] PCI host bridge to bus 0000:00 [ 0.455025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.457035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.462039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.465024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.468028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.470026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.473254] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.476289] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.479578] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.490954] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.496107] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.501022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.502013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.504010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.506697] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.508744] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.511027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.513783] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.518019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.527905] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.533014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.540708] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.548019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.567025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.610019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.704384] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.712018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.721021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.741023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.756309] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.768026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.775078] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.848018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.861106] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.869065] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.876019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.918019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.999000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.018026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.035022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.098021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.119334] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.127022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.134019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.156023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.177737] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.182025] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.187031] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.191981] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.195247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.203258] iommu: Default domain type: Passthrough [ 1.208560] SCSI subsystem initialized [ 1.211435] ACPI: bus type USB registered [ 1.214336] usbcore: registered new interface driver usbfs [ 1.217162] usbcore: registered new interface driver hub [ 1.220102] usbcore: registered new device driver usb [ 1.223303] pps_core: LinuxPPS API ver. 1 registered [ 1.226012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.231433] PTP clock support registered [ 1.235453] EDAC MC: Ver: 3.0.0 [ 1.238249] PCI: Using ACPI for IRQ routing [ 1.241926] NetLabel: Initializing [ 1.244012] NetLabel: domain hash size = 128 [ 1.247015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.251223] NetLabel: unlabeled traffic allowed by default [ 1.256147] vgaarb: loaded [ 1.259299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.263014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.307000] clocksource: Switched to clocksource kvm-clock [ 1.470935] VFS: Disk quotas dquot_6.6.0 [ 1.473421] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.477637] *** VALIDATE ramfs *** [ 1.479722] *** VALIDATE hugetlbfs *** [ 1.526894] pnp: PnP ACPI init [ 1.540960] pnp: PnP ACPI: found 6 devices [ 1.559889] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.565941] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.570504] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.572828] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.575210] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.577787] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.580671] NET: Registered protocol family 2 [ 1.583816] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.590644] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.594741] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.600540] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.604244] TCP: Hash tables configured (established 65536 bind 65536) [ 1.607513] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.610734] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.614371] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.617467] NET: Registered protocol family 1 [ 1.623023] RPC: Registered named UNIX socket transport module. [ 1.627803] RPC: Registered udp transport module. [ 1.630955] RPC: Registered tcp transport module. [ 1.634487] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.639510] NET: Registered protocol family 44 [ 1.643510] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.653840] pci 0000:00:00.0: quirk_natoma+0x0/0x20 took 10089 usecs [ 1.662339] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.668286] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.675232] PCI: CLS 0 bytes, default 64 [ 1.680553] Unpacking initramfs... [ 4.372369] debug: unmapping init [mem 0xffff8beffcc54000-0xffff8beffffbffff] [ 4.577856] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.582729] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.586533] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 7.107423] Initialise system trusted keyrings [ 7.108839] Key type blacklist registered [ 7.110476] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.119579] zbud: loaded [ 7.122509] *** VALIDATE nfs *** [ 7.123527] *** VALIDATE nfs4 *** [ 7.124842] pstore: using deflate compression [ 7.128502] Platform Keyring initialized [ 7.330331] NET: Registered protocol family 38 [ 7.331773] Key type asymmetric registered [ 7.332981] Asymmetric key parser 'x509' registered [ 7.334592] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.337272] io scheduler mq-deadline registered [ 7.338625] io scheduler kyber registered [ 7.339960] io scheduler bfq registered [ 7.341574] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.344302] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.346616] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.349273] ACPI: Power Button [PWRF] [ 7.354870] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.361116] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.628344] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.638534] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.664446] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.706128] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.738872] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.775949] Non-volatile memory driver v1.3 [ 7.781588] Linux agpgart interface v0.103 [ 7.836133] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 7.840478] vda: detected capacity change from 0 to 76681216 [ 7.879919] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.883779] vdb: detected capacity change from 0 to 1073741824 [ 8.097751] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.113021] vdc: detected capacity change from 0 to 2621440000 [ 8.142281] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.146289] vdd: detected capacity change from 0 to 2621440000 [ 8.160950] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.164792] vde: detected capacity change from 0 to 4294967296 [ 8.189426] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.192874] vdf: detected capacity change from 0 to 4294967296 [ 8.204985] libphy: Fixed MDIO Bus: probed [ 8.210979] usbcore: registered new interface driver usbserial_generic [ 8.213527] usbserial: USB Serial support registered for generic [ 8.215859] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.220817] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.222359] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.224518] mousedev: PS/2 mouse device common for all mice [ 8.227710] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.230806] rtc_cmos 00:05: RTC can wake from S4 [ 8.232954] rtc_cmos 00:05: registered as rtc0 [ 8.236592] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.240397] intel_pstate: CPU model not supported [ 8.243400] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.247106] hid: raw HID events driver (C) Jiri Kosina [ 8.252773] usbcore: registered new interface driver usbhid [ 8.254766] usbhid: USB HID core driver [ 8.257543] drop_monitor: Initializing network drop monitor service [ 8.260870] Initializing XFRM netlink socket [ 8.262901] NET: Registered protocol family 10 [ 8.266337] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.269533] Segment Routing with IPv6 [ 8.272231] NET: Registered protocol family 17 [ 8.274265] mpls_gso: MPLS GSO support [ 8.278901] RAS: Correctable Errors collector initialized. [ 8.280855] AVX version of gcm_enc/dec engaged. [ 8.282475] AES CTR mode by8 optimization enabled [ 8.567618] sched_clock: Marking stable (8567550181, 0)->(10755314082, -2187763901) [ 8.570874] registered taskstats version 1 [ 8.573148] Loading compiled-in X.509 certificates [ 8.574842] zswap: loaded using pool lzo/zbud [ 9.102795] Key type big_key registered [ 9.182879] Key type encrypted registered [ 9.184664] ima: No TPM chip found, activating TPM-bypass! [ 9.187197] ima: Allocated hash algorithm: sha1 [ 9.190712] ima: No architecture policies found [ 9.192847] evm: Initialising EVM extended attributes: [ 9.194660] evm: security.selinux [ 9.195829] evm: security.ima [ 9.197102] evm: security.capability [ 9.198552] evm: HMAC attrs: 0x1 [ 9.201618] rtc_cmos 00:05: setting system clock to 2026-09-14 06:14:06 UTC (1789366446) [ 9.208683] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 9.271432] debug: unmapping init [mem 0xffffffffb3382000-0xffffffffb3658fff] [ 9.280650] Write protecting the kernel read-only data: 28672k [ 9.284392] debug: unmapping init [mem 0xffffffffb1a03000-0xffffffffb1bfffff] [ 9.287399] debug: unmapping init [mem 0xffffffffb2314000-0xffffffffb23fffff] [ 9.334331] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.408679] systemd[1]: Detected virtualization kvm. [ 9.410809] systemd[1]: Detected architecture x86-64. [ 9.412845] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.459526] systemd[1]: No hostname configured. [ 9.461403] systemd[1]: Set hostname to . [ 9.463430] random: systemd: uninitialized urandom read (16 bytes read) [ 9.465793] systemd[1]: Initializing machine ID from random generator. [ 9.596668] random: ln: uninitialized urandom read (6 bytes read) [ 9.778260] random: systemd: uninitialized urandom read (16 bytes read) [ 9.780733] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.785169] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 9.792230] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. [ 12.127166] hrtimer: interrupt took 5113094 ns Starting dracut pre-udev hook... [ 12.462430] device-mapper: uevent: version 1.0.3 [ 12.464510] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 16.870727] random: fast init done [ 17.465730] virtio_net virtio0 ens2: renamed from eth0 [ 18.040234] scsi host0: ata_piix [ 18.429180] scsi host1: ata_piix [ 18.433871] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 18.438198] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 23.040799] random: crng init done [ 23.047624] random: 7 urandom warning(s) missed due to ratelimiting [ 26.222760] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 28.434214] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.065689] printk: systemd: 26 output lines suppressed due to ratelimiting [ 32.486833] SELinux: Disabled at runtime. [ 32.567222] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.576617] systemd[1]: Detected virtualization kvm. [ 32.577734] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 33.591804] systemd[1]: initrd-switch-root.service: Succeeded. [ 33.595914] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 33.609603] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 33.614705] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 33.619157] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 33.629213] systemd[1]: Starting Journal Service... Starting Journal Service... [ 33.641074] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK [[ 33.741898] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 35.095695] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 36.541411] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.597514] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 37.527961] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.486441] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 42.284414] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 42.830142] NFS: Registering the id_resolver key type [ 42.832127] Key type id_resolver registered [ 42.833448] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg114-server login: [ 73.249821] libcfs: loading out-of-tree module taints kernel. [ 73.267122] Key type ._llcrypt registered [ 73.268404] Key type .llcrypt registered [ 73.309487] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 81.381289] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 81.973192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.980389] alg: No test for adler32 (adler32-zlib) [ 82.996216] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 83.320492] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 84.943198] Key type lgssc registered [ 85.653476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.783345] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.918300] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 135.597597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 135.638345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.019782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 137.082974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 137.209908] Lustre: lustre-MDT0000: new disk, initializing [ 137.372874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.424908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.294973] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.540982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.737566] Lustre: 6484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.944272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.947886] Lustre: Skipped 1 previous similar message [ 170.130117] Lustre: lustre-MDT0001: new disk, initializing [ 170.257965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.324470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.340696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.799792] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.853256] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.910804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.217224] Lustre: lustre-OST0000: new disk, initializing [ 191.223432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.232524] Lustre: 8423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.318746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.316388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.330038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.435436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.580581] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.111454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.251633] Lustre: lustre-OST0001: new disk, initializing [ 211.253742] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.257251] Lustre: 9494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.303842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.988563] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.632877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.647099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.744653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.419477] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.921207] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 239.557993] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.689293] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing yml_node [ 248.627545] Lustre: DEBUG MARKER: Client: 2.17.58.6 [ 250.904160] Lustre: DEBUG MARKER: MDS: 2.17.58.6 [ 253.492279] Lustre: DEBUG MARKER: OSS: 2.17.58.6 [ 254.926739] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 02:18:10 EDT 2026 [ 271.244375] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 272.634318] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.224673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 288.236233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.252545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.760406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.761568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.114211] Lustre: server umount lustre-MDT0000 complete [ 297.221421] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789366734 with bad export cookie 4862883747610162430 [ 297.228405] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.229303] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 300.000158] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.006641] LustreError: 9461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.015501] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.015790] Lustre: Skipped 3 previous similar messages [ 300.020691] Lustre: Skipped 3 previous similar messages [ 300.048198] LustreError: 9461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 303.010725] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.012741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.032033] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 303.036353] Lustre: Skipped 2 previous similar messages [ 303.620338] Lustre: server umount lustre-MDT0001 complete [ 307.541532] Lustre: server umount lustre-OST0000 complete [ 311.905840] Lustre: server umount lustre-OST0001 complete [ 319.521723] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 328.512280] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 362.791284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.104564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 363.156712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 363.234266] Lustre: lustre-MDT0000: new disk, initializing [ 363.403939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 363.433356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 367.792117] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 378.911965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.110610] Lustre: 15048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 379.169617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 379.172714] Lustre: Skipped 1 previous similar message [ 379.261387] Lustre: lustre-MDT0001: new disk, initializing [ 379.350878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.404876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.420540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 384.334359] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 394.559537] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 400.052719] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 407.384172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 407.508223] Lustre: lustre-OST0000: new disk, initializing [ 407.509838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 407.516198] Lustre: 17018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 407.589376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.596123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.604141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.763051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.083491] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 425.317831] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 430.306975] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 430.542677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.942878] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 435.135152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 440.300701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.315599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.335462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 443.869038] Lustre: server umount lustre-OST0000 complete [ 451.040741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 451.048613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.058330] LustreError: Skipped 1 previous similar message [ 451.062070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 451.082875] Lustre: Skipped 2 previous similar messages [ 451.113949] Lustre: Skipped 2 previous similar messages [ 454.203197] Lustre: server umount lustre-MDT0000 complete [ 456.162303] LustreError: 15059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.174449] LustreError: 15059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 457.875363] LustreError: 15040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789366895 with bad export cookie 4862883747610163963 [ 457.885051] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 457.886207] LustreError: 15040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 458.282869] Lustre: server umount lustre-MDT0001 complete [ 462.485598] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:21:37 (1789366897) [ 464.788739] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 466.496474] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:21:42 (1789366902) [ 473.130281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.626405] LustreError: 19428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.687496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.578895] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 478.714326] LustreError: 19429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.894503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.639411] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 498.682257] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 503.982967] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 511.738303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 511.919048] LustreError: 21249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.955082] LustreError: 21249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 519.219138] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 526.745827] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 533.989392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.998539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.009034] Lustre: Skipped 1 previous similar message [ 534.009459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 534.014774] Lustre: Skipped 1 previous similar message [ 540.076849] Lustre: server umount lustre-OST0000 complete [ 548.833159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 550.864867] Lustre: server umount lustre-MDT0000 complete [ 554.468322] LustreError: 19424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.477334] LustreError: 19424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 554.792815] LustreError: 19408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789366992 with bad export cookie 4862883747610164845 [ 554.796076] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.799886] LustreError: 19408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 555.181760] Lustre: server umount lustre-MDT0001 complete [ 567.180388] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 570.282955] Key type lgssc unregistered [ 570.593061] LNet: 22960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.600213] LNetError: 22960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.626694] LNet: Removed LNI 192.168.201.114@tcp [ 571.791189] Key type .llcrypt unregistered [ 571.794522] Key type ._llcrypt unregistered [ 593.284030] Key type ._llcrypt registered [ 593.287355] Key type .llcrypt registered [ 593.406332] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 594.510890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 594.642027] alg: No test for adler32 (adler32-zlib) [ 595.692402] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 595.882326] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 597.495186] Key type lgssc registered [ 598.494869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 606.204299] loop: module loaded [ 606.269137] loop0: detected capacity change from 0 to 204800000 [ 606.502085] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 606.512382] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 606.840125] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 612.616297] loop0: detected capacity change from 0 to 204800000 [ 612.785366] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 612.795431] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 612.804726] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.180555] loop0: detected capacity change from 0 to 204800000 [ 623.376407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 623.405840] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.614745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 624.675715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 624.759541] Lustre: lustre-MDT0000: new disk, initializing [ 624.876590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.899209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 629.119739] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 637.649579] loop1: detected capacity change from 0 to 204800000 [ 642.742678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 642.986152] Lustre: lustre-OST07c6: new disk, initializing [ 642.990393] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 642.994761] Lustre: Skipped 1 previous similar message [ 642.999552] Lustre: 26146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 643.062028] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 649.659538] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 651.925816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 651.940366] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 651.985881] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 654.176956] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 654.183909] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 654.191045] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 654.196232] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 654.201388] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 654.205671] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 671.835136] Lustre: 26617:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 671.844187] Lustre: 26617:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 671.847730] Lustre: 26617:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 671.853630] Lustre: 26617:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 671.855983] Lustre: 26617:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 671.859319] Lustre: 26617:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 676.571077] Lustre: 25134:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 676.576675] Lustre: 25134:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 676.581795] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 676.587131] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.592818] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 676.598482] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.604429] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 676.612306] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.623288] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 676.632210] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.650298] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 676.657131] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.879596] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 679.902180] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 679.917402] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 679.928838] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.933893] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 679.943829] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.951726] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 679.960571] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.968065] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 679.974574] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.981320] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.994481] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.031681] Lustre: 25134:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 685.045616] Lustre: 25134:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 685.061985] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 685.073166] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.079363] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 685.084558] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.092246] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 685.101285] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.110164] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 685.120010] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.128527] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 685.135132] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.791090] Lustre: 25134:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 694.805245] Lustre: 25134:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 694.819569] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 694.828743] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 694.836868] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 694.845559] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 694.853477] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 694.864280] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 694.874312] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 694.889710] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 694.896902] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 694.902987] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.908280] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.926662] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 748.484230] Lustre: server umount lustre-OST07c6 complete [ 757.046548] Lustre: server umount lustre-MDT0000 complete [ 770.816936] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 773.263843] Key type lgssc unregistered [ 773.509681] LNet: 28703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.519193] LNetError: 28703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.543621] LNet: Removed LNI 192.168.201.114@tcp [ 774.277425] Key type .llcrypt unregistered [ 774.280149] Key type ._llcrypt unregistered [ 789.768290] Key type ._llcrypt registered [ 789.772499] Key type .llcrypt registered [ 789.849293] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 790.582538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 790.738431] alg: No test for adler32 (adler32-zlib) [ 791.783899] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 792.023877] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 793.687170] Key type lgssc registered [ 794.851530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.832895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.846275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.179674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.421849] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 813.545621] LustreError: 29949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.566522] LustreError: 29949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 814.196314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.413846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 817.946843] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 824.721414] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 829.549033] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 835.010081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 835.330654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 839.720695] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 845.794920] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 848.618312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 864.224867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 864.227719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.247135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 864.253852] Lustre: Skipped 1 previous similar message [ 867.569762] Lustre: server umount lustre-OST0000 complete [ 870.889530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 870.903850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.912524] Lustre: Skipped 1 previous similar message [ 870.924887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 874.467115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 874.468994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.973311] Lustre: server umount lustre-MDT0000 complete [ 879.585991] LustreError: 29944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.596255] LustreError: 29944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 880.028316] LustreError: 29929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367317 with bad export cookie 15393485661133841529 [ 880.029955] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 880.036079] LustreError: 29929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 880.329843] Lustre: server umount lustre-MDT0001 complete [ 887.859751] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 889.848203] Key type lgssc unregistered [ 890.081222] LNet: 33681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.087406] LNetError: 33681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.106409] LNet: Removed LNI 192.168.201.114@tcp [ 890.677447] Key type .llcrypt unregistered [ 890.684645] Key type ._llcrypt unregistered [ 904.905904] Key type ._llcrypt registered [ 904.907817] Key type .llcrypt registered [ 904.980886] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 905.924654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.964488] alg: No test for adler32 (adler32-zlib) [ 906.960239] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 907.139227] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 908.767150] Key type lgssc registered [ 909.714594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.809156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.827593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.137430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.730156] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 927.418428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.687138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 930.814856] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 937.209196] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 941.750552] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 947.265716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.537425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.207909] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 959.419578] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 963.956546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 973.282978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 973.295990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.312514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.305163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 974.322403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.323314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.341227] Lustre: Skipped 1 previous similar message [ 978.795738] Lustre: server umount lustre-OST0000 complete [ 984.032480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 984.040224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.057186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.576284] Lustre: server umount lustre-MDT0000 complete [ 989.668212] LustreError: 36767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.681372] LustreError: 36767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 991.579155] LustreError: 35664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367428 with bad export cookie 8039820540704012815 [ 991.585394] LustreError: 35664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 991.587241] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 991.940160] Lustre: server umount lustre-MDT0001 complete [ 1000.368811] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1002.701985] Key type lgssc unregistered [ 1002.954150] LNet: 38481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.976682] LNetError: 38481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.995909] LNet: Removed LNI 192.168.201.114@tcp [ 1003.764869] Key type .llcrypt unregistered [ 1003.767691] Key type ._llcrypt unregistered [ 1021.601401] Key type ._llcrypt registered [ 1021.605313] Key type .llcrypt registered [ 1021.772248] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1022.417820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.432432] alg: No test for adler32 (adler32-zlib) [ 1023.442751] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1023.652827] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1025.312053] Key type lgssc registered [ 1026.136194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1032.164430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1032.178043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1033.603444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.658438] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1042.919342] LustreError: 39875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.945533] LustreError: 39875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1044.122327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.412140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.526680] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1054.702732] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1059.337383] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1064.843718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1065.049665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.724724] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1075.890220] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1078.391167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1087.854255] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 1088.994605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1088.995369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.009731] Lustre: Skipped 2 previous similar messages [ 1089.015716] Lustre: Skipped 2 previous similar messages [ 1092.026506] Lustre: server umount lustre-MDT0000 complete [ 1092.954358] LustreError: 39871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.122664] LustreError: 40604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.135053] LustreError: 40604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1095.081372] LustreError: 39856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367532 with bad export cookie 5816989511792224505 [ 1095.092811] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1098.070666] LustreError: 39872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.102170] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 1099.233477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.259576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1100.266060] Lustre: Skipped 1 previous similar message [ 1101.423907] Lustre: server umount lustre-MDT0001 complete [ 1121.366337] Lustre: server umount lustre-OST0000 complete [ 1135.463233] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1137.753838] Key type lgssc unregistered [ 1137.982810] LNet: 43733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.990950] LNetError: 43733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.006932] LNet: Removed LNI 192.168.201.114@tcp [ 1138.668436] Key type .llcrypt unregistered [ 1138.670575] Key type ._llcrypt unregistered [ 1159.225382] Key type ._llcrypt registered [ 1159.228029] Key type .llcrypt registered [ 1159.343938] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1160.390763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1160.404960] alg: No test for adler32 (adler32-zlib) [ 1161.361853] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1161.527537] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1163.151876] Key type lgssc registered [ 1164.039288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1169.804576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1169.825104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.112909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.141169] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1179.779573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.976563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1182.889902] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1187.654799] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1191.522592] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1196.565610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1196.819388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.675045] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1205.829558] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1216.300885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1221.600540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1221.604578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.610056] Lustre: Skipped 1 previous similar message [ 1221.610448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.626179] Lustre: Skipped 1 previous similar message [ 1222.510303] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 1226.722153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1226.729835] Lustre: Skipped 1 previous similar message [ 1227.265163] Lustre: server umount lustre-OST0000 complete [ 1246.692656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1246.704276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.717793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1248.731740] Lustre: server umount lustre-MDT0000 complete [ 1251.469558] LustreError: 45106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367688 with bad export cookie 12821146429123451743 [ 1251.476568] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1251.480491] LustreError: 45106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1257.841725] Lustre: server umount lustre-MDT0001 complete [ 1265.596341] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1267.034806] Key type lgssc unregistered [ 1267.211860] LNet: 48850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.216143] LNetError: 48850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.232246] LNet: Removed LNI 192.168.201.114@tcp [ 1267.687271] Key type .llcrypt unregistered [ 1267.688919] Key type ._llcrypt unregistered [ 1282.118049] Key type ._llcrypt registered [ 1282.119246] Key type .llcrypt registered [ 1282.173630] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1282.830185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1282.939876] alg: No test for adler32 (adler32-zlib) [ 1283.868444] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1284.033121] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1285.655170] Key type lgssc registered [ 1286.427201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1291.732967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1291.759963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.013905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.442992] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1303.445842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.523594] LustreError: 50243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.538963] LustreError: 50243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1303.715698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1306.910717] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1312.716608] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1317.110356] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1321.882488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1322.113293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.758600] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1331.589811] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1338.542113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1346.512877] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1348.670048] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1360.063989] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-14 2h36m35s [ 1361.051831] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1363.818938] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 1364.451573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.456926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.457695] Lustre: Skipped 2 previous similar messages [ 1364.473307] Lustre: Skipped 2 previous similar messages [ 1365.538868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1365.546729] Lustre: Skipped 3 previous similar messages [ 1368.469643] Lustre: server umount lustre-MDT0000 complete [ 1369.942876] LustreError: 51345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.757272] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1371.622694] LustreError: 50242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.635877] LustreError: 50242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1375.315403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.409527] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.554623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.608544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1376.627959] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1378.299240] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1380.501376] Lustre: DEBUG MARKER: Wait for df (36658) ... [ 1381.486169] Lustre: DEBUG MARKER: done [ 1384.927958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1384.933113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.939282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1385.951937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1385.958168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.602467] Lustre: server umount lustre-OST0000 complete [ 1392.096076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1392.099127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.111750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.122886] Lustre: Skipped 4 previous similar messages [ 1397.620870] Lustre: server umount lustre-MDT0000 complete [ 1398.240851] LustreError: 50243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.246686] LustreError: 50243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1399.513981] LustreError: 50222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367836 with bad export cookie 9498071361297671714 [ 1399.518271] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.704588] Lustre: server umount lustre-MDT0001 complete [ 1405.917768] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1407.481723] Key type lgssc unregistered [ 1407.695874] LNet: 55183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.703427] LNetError: 55183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.745296] LNet: Removed LNI 192.168.201.114@tcp [ 1409.262769] Key type .llcrypt unregistered [ 1409.264586] Key type ._llcrypt unregistered [ 1434.049104] Key type ._llcrypt registered [ 1434.050885] Key type .llcrypt registered [ 1434.137383] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1434.971413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1434.994614] alg: No test for adler32 (adler32-zlib) [ 1435.945785] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1436.142062] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1437.783158] Key type lgssc registered [ 1438.497640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1443.479950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1443.499184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.676253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1444.693984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1444.826367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.407419] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1452.554344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.633656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1452.651818] Lustre: 57366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1452.800154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1455.100802] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1459.218204] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1462.775960] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1466.965834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.083430] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1467.196980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.463131] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1474.623212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1474.753701] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1478.845259] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-14 2h38m35s [ 1479.685554] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1483.547442] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1484.811716] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1484.840525] LustreError: 57375:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1505.128349] Lustre: lustre-MDT0000: Client 215c7f79-55d6-4961-b540-bb4cd0ef39ce (at 192.168.201.14@tcp) reconnecting [ 1524.895120] LustreError: 57375:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1526.184742] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1527.523239] Lustre: DEBUG MARKER: done [ 1530.850750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1530.851663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.854769] LustreError: Skipped 1 previous similar message [ 1530.858775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1530.860623] Lustre: Skipped 1 previous similar message [ 1535.983444] Lustre: server umount lustre-OST0000 complete [ 1540.065876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1540.074461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.080472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1540.084749] Lustre: Skipped 1 previous similar message [ 1541.154768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1541.157584] Lustre: Skipped 2 previous similar messages [ 1543.857976] Lustre: server umount lustre-MDT0000 complete [ 1545.842483] LustreError: 57360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789367983 with bad export cookie 6711312909959493168 [ 1545.844134] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1545.850014] LustreError: 57360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1546.101284] Lustre: server umount lustre-MDT0001 complete [ 1551.455610] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1553.095272] Key type lgssc unregistered [ 1553.297908] LNet: 61690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.301684] LNetError: 61690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.310851] LNet: Removed LNI 192.168.201.114@tcp [ 1553.843134] Key type .llcrypt unregistered [ 1553.845035] Key type ._llcrypt unregistered [ 1577.869981] Key type ._llcrypt registered [ 1577.871297] Key type .llcrypt registered [ 1577.953300] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1578.635356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1578.646116] alg: No test for adler32 (adler32-zlib) [ 1579.569316] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1579.694502] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1581.295138] Key type lgssc registered [ 1581.891528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1583.687214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1583.695568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.893167] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1584.918841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1585.019043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.584829] Lustre: Failing over lustre-MDT0000 [ 1587.741302] Lustre: server umount lustre-MDT0000 complete [ 1591.645520] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:40:27 (1789368027) [ 1595.447278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.715068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.807333] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1603.959875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.050940] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1604.078586] Lustre: 64459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1604.282679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1606.749771] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1611.074330] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1614.287249] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1618.676724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1618.807174] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1618.982927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.817363] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1625.664883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1627.868795] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1632.793694] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1640.417518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1640.426497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.446238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.930221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.951622] Lustre: Skipped 1 previous similar message [ 1640.965458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.967323] Lustre: Skipped 1 previous similar message [ 1642.707078] Lustre: server umount lustre-MDT0000 complete [ 1645.458332] LustreError: 64452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368082 with bad export cookie 2412369191467573900 [ 1645.460364] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1645.465309] LustreError: 64452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1645.645047] Lustre: server umount lustre-MDT0001 complete [ 1647.954191] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1651.025417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1656.905317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.432339] Lustre: 63384:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368083/real 1789368083] req@000000008af28edc x1876288356779648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1662.448812] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.463661] Lustre: 63385:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368087/real 1789368087] req@00000000fc6470e4 x1876288356779776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1667.551166] Lustre: 63386:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368088/real 1789368088] req@000000002dab1f2f x1876288356779904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1670.687959] LustreError: 63383:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fc6470e4 x1876288356780800/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1670.953090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.026813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1671.972270] Lustre: 63386:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368092/real 1789368092] req@000000008457a569 x1876288356780160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1673.641516] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1676.063281] Lustre: 63384:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368097/real 1789368097] req@00000000765fa6f0 x1876288356780288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1677.281439] LustreError: 68229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.291294] LustreError: 68229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1679.727567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.951121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1681.007819] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1683.151227] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1688.279875] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1691.521788] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1696.739656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.750640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.752258] Lustre: Skipped 1 previous similar message [ 1701.623585] Lustre: server umount lustre-MDT0000 complete [ 1701.857451] LustreError: 66307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1701.874698] LustreError: 66307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1703.849478] LustreError: 64452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368141 with bad export cookie 2412369191467576175 [ 1703.851271] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1703.858039] LustreError: 64452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1704.024894] Lustre: server umount lustre-MDT0001 complete [ 1707.640602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1712.915255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1714.015262] LustreError: 70894:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1714.024607] LustreError: 70894:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000027851a60 x1876288356803328/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789368151 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1714.039023] LustreError: 70894:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1714.343359] LustreError: 66307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.407788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.451081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1717.319972] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1719.777773] LustreError: 66308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.778556] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.796673] Lustre: Skipped 1 previous similar message [ 1722.746332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.849352] Lustre: 63384:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789368144/real 1789368144] req@0000000077006183 x1876288356803584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789368160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1722.873647] Lustre: 63384:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1725.360651] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1727.086828] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1727.091615] Lustre: Skipped 1 previous similar message [ 1729.518313] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1732.380517] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1737.697879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.700823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.706220] Lustre: Skipped 2 previous similar messages [ 1737.708583] Lustre: Skipped 3 previous similar messages [ 1742.003178] Lustre: server umount lustre-MDT0000 complete [ 1742.816233] LustreError: 70908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.825507] LustreError: 70908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1743.959090] LustreError: 65195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368181 with bad export cookie 2412369191467577554 [ 1743.960357] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.968569] LustreError: 65195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1744.143546] Lustre: server umount lustre-MDT0001 complete [ 1747.864248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1749.912432] Lustre: DEBUG MARKER: files compared the same [ 1761.527935] Lustre: server umount lustre-OST0000 complete [ 1771.004933] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1772.601124] Key type lgssc unregistered [ 1772.783513] LNet: 74178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.786494] LNetError: 74178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.798648] LNet: Removed LNI 192.168.201.114@tcp [ 1773.238466] Key type .llcrypt unregistered [ 1773.239973] Key type ._llcrypt unregistered [ 1786.236846] Key type ._llcrypt registered [ 1786.238278] Key type .llcrypt registered [ 1786.304840] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1786.878119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1786.978294] alg: No test for adler32 (adler32-zlib) [ 1787.934640] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1788.092215] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1789.719799] Key type lgssc registered [ 1790.292808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.516069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.525192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.743602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.700824] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 1801.904701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.063066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.622837] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 1808.733685] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1812.646848] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1816.499165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.707872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.852634] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug +malloc all [ 1820.841531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1823.803750] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1834.851836] Lustre: server umount lustre-OST0000 complete [ 1837.538340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1837.542305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1837.546223] Lustre: Skipped 1 previous similar message [ 1837.556230] Lustre: Skipped 1 previous similar message [ 1842.655751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.661523] Lustre: Skipped 1 previous similar message [ 1846.831416] Lustre: server umount lustre-MDT0000 complete [ 1847.781485] LustreError: 75569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.790587] LustreError: 75569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1848.687351] LustreError: 75549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368285 with bad export cookie 3920594700858910926 [ 1848.689060] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1848.691960] LustreError: 75549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1857.903873] Lustre: server umount lustre-MDT0001 complete [ 1863.785790] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1865.466624] Key type lgssc unregistered [ 1865.638895] LNet: 79097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.643189] LNetError: 79097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.654462] LNet: Removed LNI 192.168.201.114@tcp [ 1866.081765] Key type .llcrypt unregistered [ 1866.082899] Key type ._llcrypt unregistered [ 1879.902446] Key type ._llcrypt registered [ 1879.904214] Key type .llcrypt registered [ 1879.973180] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 1880.644577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1880.658094] alg: No test for adler32 (adler32-zlib) [ 1881.572106] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 1881.709753] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 1883.311256] Key type lgssc registered [ 1883.983981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1888.365831] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1888.377920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1903.967431] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1909.089606] LustreError: 80462:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.114@tcp: failed processing log, type 4: rc = -110 [ 1935.711815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.554533] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1944.371635] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1949.504379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.584707] Lustre: *** cfs_fail_loc=706, val=0*** [ 1949.837158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.895420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1952.652637] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1956.322704] LustreError: 80488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.335646] LustreError: 80488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1957.401566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.590384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1959.445206] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 1963.099033] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1965.972953] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1968.102666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.102987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1968.117552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1968.122754] Lustre: Skipped 1 previous similar message [ 1973.155269] Lustre: server umount lustre-OST0000 complete [ 1978.336669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.337750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.340667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1978.344593] Lustre: Skipped 2 previous similar messages [ 1981.488081] Lustre: server umount lustre-MDT0000 complete [ 1983.456412] LustreError: 81544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.469257] LustreError: 81544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1983.690361] LustreError: 80468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368420 with bad export cookie 14343625588122633754 [ 1983.691146] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.695952] LustreError: 80468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1983.942243] Lustre: server umount lustre-MDT0001 complete [ 1990.203337] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 1991.831393] Key type lgssc unregistered [ 1992.018440] LNet: 84072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.023848] LNetError: 84072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.036906] LNet: Removed LNI 192.168.201.114@tcp [ 1992.643077] Key type .llcrypt unregistered [ 1992.644686] Key type ._llcrypt unregistered [ 2006.392210] Key type ._llcrypt registered [ 2006.393759] Key type .llcrypt registered [ 2006.447642] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2006.889357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2006.901809] alg: No test for adler32 (adler32-zlib) [ 2007.817950] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2007.992850] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2009.631106] Key type lgssc registered [ 2010.116470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2013.894326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2013.920582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.994299] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2020.976229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.244377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.103847] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2026.464327] LustreError: 86149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.480764] LustreError: 86149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2027.318979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2027.543408] LustreError: 86148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.557985] LustreError: 86148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2027.568554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.839562] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2031.654608] LustreError: 86152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.977668] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2034.983658] LustreError: 84820:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000bcff4e2 x1876288806593024/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2034.998870] LustreError: 86153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.999818] Lustre: MGS: Client ec8bce7f-0b8a-4660-851a-79e73b3aa62e (at 0@lo) reconnecting [ 2035.004452] LustreError: 86153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2035.162596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.210310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2037.367554] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2047.459081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.466222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.471451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2050.530981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2050.532606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2050.537462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.550442] Lustre: Skipped 1 previous similar message [ 2053.461597] Lustre: server umount lustre-OST0000 complete [ 2055.653315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.661103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2055.665769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.669995] Lustre: Skipped 1 previous similar message [ 2060.768780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.773273] Lustre: Skipped 1 previous similar message [ 2061.778428] Lustre: server umount lustre-MDT0000 complete [ 2063.729165] Lustre: server umount lustre-MDT0001 complete [ 2065.606454] Lustre: server umount MGS complete [ 2071.011585] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:48:27 (1789368507) [ 2084.388179] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2088.924806] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2102.900857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.599847] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2109.330190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.434846] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2109.453610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2109.497099] Lustre: lustre-MDT0001: new disk, initializing [ 2109.527249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2109.538280] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2109.541243] Lustre: 93308:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2109.543946] LustreError: 93308:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2111.184270] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2116.419354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2116.538464] Lustre: lustre-OST0000: new disk, initializing [ 2116.541540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2116.545068] Lustre: 94075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2116.574666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.018514] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2120.095894] Lustre: 94428:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003ea12a83 x1876288806620672/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789368573 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2123.432394] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2126.496879] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.503744] Lustre: MGS: Client 6a725e1e-d75a-4c87-8332-07a32cef8887 (at 0@lo) reconnecting [ 2126.524227] Lustre: 95118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2126.533405] LustreError: 93293:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2126.537274] LustreError: 93293:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2126.541250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2126.581872] Lustre: lustre-MDT0000: new disk, initializing [ 2126.642532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.657689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2127.714865] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2127.715488] Lustre: 95133:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003802e8c1 x1876288806626816/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789368581 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2127.734388] Lustre: 95133:0:(client.c:1652:after_reply()) Skipped 1 previous similar message [ 2128.448870] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2136.808928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2136.813826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2136.831514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2161.120148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2161.123894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.128683] Lustre: Skipped 1 previous similar message [ 2161.130916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.871273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2171.877753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.880111] Lustre: Skipped 4 previous similar messages [ 2171.945198] Lustre: server umount lustre-OST0000 complete [ 2176.992417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2176.993423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.001067] Lustre: Skipped 2 previous similar messages [ 2179.658279] Lustre: server umount lustre-MDT0000 complete [ 2187.387757] Lustre: server umount lustre-MDT0001 complete [ 2189.106549] Lustre: server umount MGS complete [ 2193.544060] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:50:30 (1789368630) [ 2200.252987] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2201.509919] Key type lgssc unregistered [ 2201.676567] LNet: 97644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2201.679838] LNetError: 97644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.691619] LNet: Removed LNI 192.168.201.114@tcp [ 2202.063263] Key type .llcrypt unregistered [ 2202.064330] Key type ._llcrypt unregistered [ 2211.358255] Key type ._llcrypt registered [ 2211.359247] Key type .llcrypt registered [ 2211.429418] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2211.804220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2211.823163] alg: No test for adler32 (adler32-zlib) [ 2212.752830] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2212.891622] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2214.527177] Key type lgssc registered [ 2215.206679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2219.125516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2219.133973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.377650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.519193] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2227.238722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.412616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2229.490596] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2233.530694] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.270819] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2239.588977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2239.766719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.027385] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2244.912272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2246.666436] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2255.329024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2255.332245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.340177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2257.378813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2257.381319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2257.385051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.400176] Lustre: Skipped 1 previous similar message [ 2260.838681] Lustre: server umount lustre-OST0000 complete [ 2263.520132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2263.523706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.528945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2267.621113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2267.622376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.737341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.744297] Lustre: Skipped 2 previous similar messages [ 2277.345423] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2277.423813] Lustre: server umount lustre-MDT0000 complete [ 2278.934273] LustreError: 98822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368716 with bad export cookie 8814776429894745120 [ 2278.936843] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2278.942260] LustreError: 98822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2279.065075] Lustre: server umount lustre-MDT0001 complete [ 2283.509265] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2284.751696] Key type lgssc unregistered [ 2284.883395] LNet: 102369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.887066] LNetError: 102369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.895465] LNet: Removed LNI 192.168.201.114@tcp [ 2285.335826] Key type .llcrypt unregistered [ 2285.337134] Key type ._llcrypt unregistered [ 2293.251203] Key type ._llcrypt registered [ 2293.252750] Key type .llcrypt registered [ 2293.294595] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2293.733350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2293.759451] alg: No test for adler32 (adler32-zlib) [ 2294.651438] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2294.784632] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2296.391196] Key type lgssc registered [ 2297.001930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2300.192910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2300.196918] LustreError: 103419:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2300.225265] LustreError: 103425:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2300.229995] LustreError: 103419:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4968 [ 2300.233385] LustreError: 103419:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2300.237813] LustreError: 103419:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2300.248863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.479539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2305.467349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.604505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2307.668490] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2310.356533] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing lsmod [ 2311.867600] LustreError: 104629:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2311.878409] LustreError: 104632:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2311.882344] LustreError: 104629:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 2311.887049] LustreError: 104629:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2311.891274] LustreError: 104629:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2311.891433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2312.072475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.491107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.496281] Lustre: Skipped 1 previous similar message [ 2315.498610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2315.501364] Lustre: Skipped 1 previous similar message [ 2319.595180] Lustre: server umount lustre-MDT0000 complete [ 2320.867336] LustreError: 103457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.873586] LustreError: 103457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2321.183333] LustreError: 103444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368758 with bad export cookie 10142261243435046877 [ 2321.185194] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2321.188047] LustreError: 103444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2321.591243] LustreError: 103893:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a9025d64 x1876289107543680/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2321.654894] Lustre: server umount lustre-MDT0001 complete [ 2333.226245] Lustre: server umount lustre-OST0000 complete [ 2337.762813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.991194] LustreError: 105699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.028289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.016862] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2343.397319] LustreError: 105700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.405715] LustreError: 105700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2344.508536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.699713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.753452] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2350.941466] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2353.283804] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2356.374925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2356.590869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.085849] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2362.574218] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2362.729995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2373.093252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2373.097792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.103793] Lustre: Skipped 1 previous similar message [ 2373.105799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2373.108092] Lustre: Skipped 1 previous similar message [ 2374.112731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.115319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.121827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.317844] Lustre: server umount lustre-OST0000 complete [ 2379.233200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.233820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2379.240050] Lustre: Skipped 1 previous similar message [ 2379.242123] Lustre: Skipped 1 previous similar message [ 2384.356804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.365794] Lustre: Skipped 2 previous similar messages [ 2385.234690] Lustre: server umount lustre-MDT0000 complete [ 2387.061178] LustreError: 105681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368824 with bad export cookie 10142261243435047563 [ 2387.063955] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.066897] LustreError: 105681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2387.223192] Lustre: server umount lustre-MDT0001 complete [ 2392.644559] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2394.029664] Key type lgssc unregistered [ 2394.210208] LNet: 109228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2394.213845] LNetError: 109228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.223562] LNet: Removed LNI 192.168.201.114@tcp [ 2394.644902] Key type .llcrypt unregistered [ 2394.645995] Key type ._llcrypt unregistered [ 2406.255636] Key type ._llcrypt registered [ 2406.257271] Key type .llcrypt registered [ 2406.305278] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2406.677709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.717905] alg: No test for adler32 (adler32-zlib) [ 2407.613266] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2407.724487] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2409.319282] Key type lgssc registered [ 2409.840326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2413.580722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2413.590033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.783627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.371514] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2419.870492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.039712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2422.010892] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2425.152944] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2427.641310] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2431.198316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2431.362280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.844953] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2436.621880] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2440.617675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2450.026500] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2451.085879] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2455.011459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.013292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.017029] Lustre: Skipped 1 previous similar message [ 2455.017307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.019480] Lustre: Skipped 1 previous similar message [ 2458.348687] Lustre: server umount lustre-OST0000 complete [ 2460.129055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.129507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2460.133205] Lustre: Skipped 1 previous similar message [ 2465.248084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2465.250164] Lustre: Skipped 2 previous similar messages [ 2466.209725] Lustre: server umount lustre-MDT0000 complete [ 2467.848398] LustreError: 110616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368905 with bad export cookie 4900181845914203451 [ 2467.850345] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.852652] LustreError: 110616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2467.977684] Lustre: server umount lustre-MDT0001 complete [ 2472.787230] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2474.026967] Key type lgssc unregistered [ 2474.164549] LNet: 114288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.168702] LNetError: 114288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.181188] LNet: Removed LNI 192.168.201.114@tcp [ 2474.545354] Key type .llcrypt unregistered [ 2474.547561] Key type ._llcrypt unregistered [ 2483.059538] Key type ._llcrypt registered [ 2483.061141] Key type .llcrypt registered [ 2483.114735] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2483.599299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2483.722273] alg: No test for adler32 (adler32-zlib) [ 2484.608677] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2484.706721] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2486.311147] Key type lgssc registered [ 2486.789079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.295622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2490.305035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.497128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.408185] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2497.084406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.266407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2499.365785] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2503.299231] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2506.125338] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2510.026130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.231418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.101565] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2516.296740] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2516.398115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2526.689188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2526.696635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.707299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.712593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.713731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2527.714201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.717985] Lustre: Skipped 1 previous similar message [ 2530.918300] Lustre: server umount lustre-OST0000 complete [ 2532.832551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.833668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.837111] Lustre: Skipped 1 previous similar message [ 2532.841511] Lustre: Skipped 1 previous similar message [ 2537.952450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.956267] Lustre: Skipped 1 previous similar message [ 2538.714131] Lustre: server umount lustre-MDT0000 complete [ 2540.349410] LustreError: 116215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789368977 with bad export cookie 3267412868645844856 [ 2540.351672] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2540.354822] LustreError: 116215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2540.495225] Lustre: server umount lustre-MDT0001 complete [ 2544.455536] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2545.705093] Key type lgssc unregistered [ 2545.866698] LNet: 119011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2545.870588] LNetError: 119011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.885186] LNet: Removed LNI 192.168.201.114@tcp [ 2546.260625] Key type .llcrypt unregistered [ 2546.262208] Key type ._llcrypt unregistered [ 2557.945888] Key type ._llcrypt registered [ 2557.947029] Key type .llcrypt registered [ 2557.999438] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2558.502290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.517490] alg: No test for adler32 (adler32-zlib) [ 2559.409551] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2559.528169] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2561.127174] Key type lgssc registered [ 2561.594542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2565.199099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2565.206092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.448983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.254097] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2571.962742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.167485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.970330] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2577.018596] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2579.210648] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2582.205711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2582.365084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.774967] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2587.851146] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2591.592387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2595.774816] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2597.549160] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2597.552766] Lustre: Skipped 2 previous similar messages [ 2599.635994] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2599.656944] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2599.661923] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2599.665041] Lustre: Skipped 2 previous similar messages [ 2608.638456] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2608.640229] Lustre: Skipped 1 previous similar message [ 2611.244920] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.14@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2611.248924] Lustre: Skipped 2 previous similar messages [ 2616.330372] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.14@tcp 192.168.0.[2,10]@tcp [ 2621.771438] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.14/24@tcp in log lustre-MDT0000 [ 2621.775526] Lustre: Skipped 2 previous similar messages [ 2625.468793] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.14/24@tcp [ 2625.471357] Lustre: Skipped 1 previous similar message [ 2629.600238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2629.602327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.608113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.676483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.677640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2632.680883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.683829] LustreError: Skipped 2 previous similar messages [ 2634.991615] Lustre: server umount lustre-OST0000 complete [ 2637.801020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.803111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.812212] Lustre: Skipped 1 previous similar message [ 2637.826214] Lustre: Skipped 2 previous similar messages [ 2642.914224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.920012] Lustre: Skipped 1 previous similar message [ 2643.070922] Lustre: server umount lustre-MDT0000 complete [ 2644.930258] LustreError: 120380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369082 with bad export cookie 9615341397825156448 [ 2644.933381] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2644.937027] LustreError: 120380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2645.081293] Lustre: server umount lustre-MDT0001 complete [ 2650.285401] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2651.788212] Key type lgssc unregistered [ 2651.958381] LNet: 125578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2651.961485] LNetError: 125578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2653.030655] LNet: Removed LNI 192.168.201.114@tcp [ 2653.459077] Key type .llcrypt unregistered [ 2653.460829] Key type ._llcrypt unregistered [ 2666.822108] Key type ._llcrypt registered [ 2666.823398] Key type .llcrypt registered [ 2666.878385] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2667.429567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2667.441466] alg: No test for adler32 (adler32-zlib) [ 2668.323697] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2668.433516] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2670.023173] Key type lgssc registered [ 2670.486847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2674.030617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2674.047100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.267173] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2675.285052] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.14/24@tcp [ 2675.285295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.294177] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2681.755386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.904226] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2681.915670] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.14/24@tcp [ 2681.956871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2683.736253] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2687.010749] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2689.616021] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2692.770325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2692.927730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.490869] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2699.053234] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2700.075870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2715.615899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2715.619780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.624783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2717.666126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2717.667487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2717.670321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.027608] Lustre: server umount lustre-OST0000 complete [ 2722.785546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2722.787437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.794225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.804807] Lustre: Skipped 1 previous similar message [ 2722.810484] Lustre: Skipped 2 previous similar messages [ 2727.906462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2727.911988] Lustre: Skipped 1 previous similar message [ 2728.073623] Lustre: server umount lustre-MDT0000 complete [ 2729.947913] LustreError: 127091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369167 with bad export cookie 16699076578136385902 [ 2729.950577] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.951699] LustreError: 127091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2730.101502] Lustre: server umount lustre-MDT0001 complete [ 2734.978625] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2736.420786] Key type lgssc unregistered [ 2736.583245] LNet: 130717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2736.587086] LNetError: 130717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2736.598865] LNet: Removed LNI 192.168.201.114@tcp [ 2737.033856] Key type .llcrypt unregistered [ 2737.035721] Key type ._llcrypt unregistered [ 2749.755718] Key type ._llcrypt registered [ 2749.756947] Key type .llcrypt registered [ 2749.829748] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2757.866742] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2758.332473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2758.347601] alg: No test for adler32 (adler32-zlib) [ 2759.268914] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2759.414403] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2761.031165] Key type lgssc registered [ 2761.504614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2777.639786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2777.654709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.767203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2778.783759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2778.831933] Lustre: lustre-MDT0000: new disk, initializing [ 2778.882887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.899295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2780.773101] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2786.364568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.430440] Lustre: 134087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2786.477175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2786.481260] Lustre: Skipped 1 previous similar message [ 2786.522281] Lustre: lustre-MDT0001: new disk, initializing [ 2786.595789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.618556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2786.630765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2788.754838] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2793.798463] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2796.361492] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2799.559932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.683299] Lustre: lustre-OST0000: new disk, initializing [ 2799.685857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2799.689740] Lustre: 136052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2799.733891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.636985] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2807.191920] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2808.841308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2808.845355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2808.894204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2809.118515] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2809.216756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2811.007176] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2811.111757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2813.926786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.930052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2813.933959] Lustre: Skipped 1 previous similar message [ 2813.935157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2813.936268] Lustre: Skipped 1 previous similar message [ 2818.213432] Lustre: server umount lustre-OST0000 complete [ 2822.624111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2822.629844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.635260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.161030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.164585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.170082] Lustre: Skipped 1 previous similar message [ 2826.420150] Lustre: server umount lustre-MDT0000 complete [ 2828.467255] LustreError: 134078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369265 with bad export cookie 12123438557907679485 [ 2828.469927] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2828.474095] LustreError: 134078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2828.639810] Lustre: server umount lustre-MDT0001 complete [ 2832.336909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.664412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.608883] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2837.985184] LustreError: 138281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.991885] LustreError: 138281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2838.757940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.947240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2840.732262] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2843.984990] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2846.630533] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2850.233213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.422310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.515497] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2856.847456] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2858.814429] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2858.919168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2860.766976] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2863.933666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 2868.821163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.910866] Lustre: lustre-OST0001: new disk, initializing [ 2868.912883] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2868.915484] Lustre: 141670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2868.953097] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2871.426612] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2875.285319] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2877.459995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2877.465079] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2877.513770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2878.430654] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2880.000732] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2880.073818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2882.140607] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9abd50eea000.ost_server_uuid 50 [ 2882.733486] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9abd50eea000.ost_server_uuid in FULL state after 0 sec [ 2887.649893] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2887.651348] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.653807] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2887.656330] Lustre: Skipped 1 previous similar message [ 2887.660213] Lustre: Skipped 1 previous similar message [ 2891.270162] Lustre: server umount lustre-OST0001 complete [ 2892.771449] LustreError: 142311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.781518] LustreError: 142311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2893.796672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.800083] LustreError: Skipped 1 previous similar message [ 2893.801977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.807647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.890715] LustreError: 140099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.896316] LustreError: 140099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2903.009344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2903.011337] LustreError: 140101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.011721] Lustre: Skipped 4 previous similar messages [ 2907.103107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2907.151640] Lustre: server umount lustre-OST0000 complete [ 2910.687918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2910.690569] LustreError: Skipped 1 previous similar message [ 2910.692114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.696244] Lustre: Skipped 1 previous similar message [ 2914.590292] Lustre: server umount lustre-MDT0000 complete [ 2915.983973] LustreError: 138260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369353 with bad export cookie 12123438557907680360 [ 2915.987418] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2915.987806] LustreError: 138260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2916.113474] Lustre: server umount lustre-MDT0001 complete [ 2923.256963] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 2924.455231] Key type lgssc unregistered [ 2924.602384] LNet: 144688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2924.605584] LNetError: 144688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2924.613333] LNet: Removed LNI 192.168.201.114@tcp [ 2924.968990] Key type .llcrypt unregistered [ 2924.970238] Key type ._llcrypt unregistered [ 2940.772135] Key type ._llcrypt registered [ 2940.773577] Key type .llcrypt registered [ 2940.811865] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 2947.123763] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 2947.584578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2947.590490] alg: No test for adler32 (adler32-zlib) [ 2948.447159] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 2948.536577] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 2950.119181] Key type lgssc registered [ 2950.495822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2961.928402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2961.932837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.012193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2963.021470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2963.057835] Lustre: lustre-MDT0000: new disk, initializing [ 2963.077775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.083369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2964.349805] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2968.384189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.422610] Lustre: 148812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2968.444776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2968.446486] Lustre: Skipped 1 previous similar message [ 2968.482359] Lustre: lustre-MDT0001: new disk, initializing [ 2968.517148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2968.528273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2968.531743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2969.836370] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2973.486523] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2975.249079] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2977.548664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2977.643898] Lustre: lustre-OST0000: new disk, initializing [ 2977.645653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2977.648472] Lustre: 150779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2977.676483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.114924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2978.117909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2978.151546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2979.561014] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 2983.213980] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2985.143073] Lustre: Failing over lustre-OST0000 [ 2985.209389] Lustre: server umount lustre-OST0000 complete [ 2998.089953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2998.194451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.200576] Lustre: Skipped 1 previous similar message [ 2998.201255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.209053] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2999.644288] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3000.156764] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3001.448944] Lustre: Failing over lustre-MDT0000 [ 3001.537279] Lustre: server umount lustre-MDT0000 complete [ 3003.360454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.364304] Lustre: Skipped 1 previous similar message [ 3004.383125] Lustre: 147039:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789369425/real 1789369425] req@00000000fd6134d9 x1876289792273408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789369441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3004.393058] Lustre: 147039:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3008.480104] LustreError: 148821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.484979] LustreError: 148821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3009.503121] Lustre: 147041:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789369430/real 1789369430] req@00000000728166d8 x1876289792274688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789369446 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3009.882676] LustreError: 150787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.600331] LustreError: 150002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.604261] LustreError: 150002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3014.519198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.565908] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.570815] Lustre: 148814:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3014.640153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3014.658629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.686838] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3014.996125] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3016.138047] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3019.753361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3019.763777] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3019.767352] Lustre: Skipped 1 previous similar message [ 3019.784181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3024.866719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.867379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3024.870108] Lustre: Skipped 1 previous similar message [ 3024.872707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3024.875226] Lustre: Skipped 1 previous similar message [ 3026.890487] Lustre: server umount lustre-OST0000 complete [ 3029.984610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3029.985027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.987349] LustreError: Skipped 1 previous similar message [ 3029.993256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3029.996600] Lustre: Skipped 1 previous similar message [ 3034.330807] Lustre: server umount lustre-MDT0000 complete [ 3035.104060] LustreError: 148822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.109732] LustreError: 148822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3035.567082] LustreError: 148803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369472 with bad export cookie 1307122317547435232 [ 3035.570496] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3035.571950] LustreError: 148803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3035.677196] Lustre: server umount lustre-MDT0001 complete [ 3039.376828] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3040.484320] Key type lgssc unregistered [ 3040.618425] LNet: 154426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3040.621767] LNetError: 154426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3040.632318] LNet: Removed LNI 192.168.201.114@tcp [ 3040.979035] Key type .llcrypt unregistered [ 3040.980036] Key type ._llcrypt unregistered [ 3061.852838] Key type ._llcrypt registered [ 3061.854062] Key type .llcrypt registered [ 3061.899150] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3062.391871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3062.440482] alg: No test for adler32 (adler32-zlib) [ 3063.374437] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 3063.480150] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3065.072395] Key type lgssc registered [ 3065.501748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3068.440100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3068.448966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.625614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.056882] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3075.034319] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3077.066269] LustreError: 155970:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3077.070824] LustreError: 155970:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3077.074299] LustreError: 155970:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3077.212679] Lustre: server umount lustre-MDT0000 complete [ 3082.300614] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3085.470764] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3097.799441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.893090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3097.905538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3097.946775] Lustre: lustre-MDT0000: new disk, initializing [ 3097.971574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.979651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3099.343098] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3103.661365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.709224] Lustre: 159263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3103.738373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3103.741159] Lustre: Skipped 1 previous similar message [ 3103.777357] Lustre: lustre-MDT0001: new disk, initializing [ 3103.813753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3103.832232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3103.835662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3105.252694] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3109.123206] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3110.856863] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3113.146322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.249748] Lustre: lustre-OST0000: new disk, initializing [ 3113.251711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3113.254402] Lustre: 161235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3113.286531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.259531] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3119.062526] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3119.607981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3119.612851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3119.652965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3126.286587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3126.916583] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:06:03 (1789369563) [ 3206.111470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3206.114211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.119670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.623671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.623903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.629907] Lustre: Skipped 1 previous similar message [ 3211.167567] Lustre: server umount lustre-MDT0000 complete [ 3211.743473] LustreError: 161894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.747938] LustreError: 161894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3212.470436] LustreError: 161234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369649 with bad export cookie 9419704141061702802 [ 3212.471433] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3212.474035] LustreError: 161234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3216.863542] LustreError: 159272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.863596] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.874094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3216.876331] Lustre: Skipped 1 previous similar message [ 3217.887450] LustreError: 159300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.655519] Lustre: server umount lustre-MDT0001 complete [ 3234.271212] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3234.318497] Lustre: server umount lustre-OST0000 complete [ 3237.687216] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3241.424863] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3254.454102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.545821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3254.561747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3254.600480] Lustre: lustre-MDT0000: new disk, initializing [ 3254.625229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.635030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3256.088444] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3260.411068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.456484] Lustre: 165863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3260.490964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3260.494479] Lustre: Skipped 1 previous similar message [ 3260.539231] Lustre: lustre-MDT0001: new disk, initializing [ 3260.588159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3260.592074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3262.196699] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3266.240779] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3268.279755] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3270.780881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3270.893576] Lustre: lustre-OST0000: new disk, initializing [ 3270.896099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3270.900345] Lustre: 167835:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3270.938702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.941434] Lustre: Skipped 1 previous similar message [ 3273.128286] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3277.457679] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3279.119448] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3279.864525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3279.867898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3279.901646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3280.226833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3281.732464] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3281.813213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3284.961352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3284.961764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3284.964036] LustreError: Skipped 1 previous similar message [ 3284.964625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3284.968204] Lustre: Skipped 1 previous similar message [ 3284.969749] Lustre: Skipped 1 previous similar message [ 3288.594034] Lustre: server umount lustre-OST0000 complete [ 3290.080517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.081327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.085106] Lustre: Skipped 1 previous similar message [ 3290.087314] Lustre: Skipped 1 previous similar message [ 3295.951299] Lustre: server umount lustre-MDT0000 complete [ 3297.184249] LustreError: 165854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369734 with bad export cookie 9419704141061861632 [ 3297.187314] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.187468] LustreError: 165854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3297.298544] Lustre: server umount lustre-MDT0001 complete [ 3299.962266] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:08:56 (1789369736) [ 3305.106884] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3308.073249] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3319.584544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.670233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3319.682369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3319.720327] Lustre: lustre-MDT0000: new disk, initializing [ 3319.748985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.756465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3321.189177] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3325.311591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.352185] Lustre: 172586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3325.453452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3326.802874] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3330.564885] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3332.357254] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3334.905854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3335.028507] Lustre: 174565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3337.127294] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3338.232578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3338.274363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3341.084457] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3343.328532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3343.330313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.332233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.336437] Lustre: Skipped 1 previous similar message [ 3343.338757] Lustre: Skipped 2 previous similar messages [ 3349.206741] Lustre: server umount lustre-OST0000 complete [ 3351.008180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3351.012538] LustreError: Skipped 1 previous similar message [ 3356.637731] Lustre: server umount lustre-MDT0000 complete [ 3357.960134] LustreError: 173387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369795 with bad export cookie 9419704141061862500 [ 3357.962564] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.964163] LustreError: 173387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3358.087211] Lustre: server umount lustre-MDT0001 complete [ 3361.007683] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:09:57 (1789369797) [ 3366.538125] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3370.025570] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3381.978833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.062543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3382.073477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3382.075798] Lustre: Skipped 3 previous similar messages [ 3382.110770] Lustre: lustre-MDT0000: new disk, initializing [ 3382.112566] Lustre: Skipped 2 previous similar messages [ 3382.139204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.141631] Lustre: Skipped 2 previous similar messages [ 3382.147952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3382.151419] Lustre: Skipped 2 previous similar messages [ 3383.536645] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3387.975085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.021920] Lustre: 178878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3388.132495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3389.718809] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3393.771242] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3395.718074] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3398.089713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3398.197719] Lustre: 180851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3400.296509] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3404.077468] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3405.339110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3415.521266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.521749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.523588] LustreError: Skipped 1 previous similar message [ 3415.524128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.528586] Lustre: Skipped 3 previous similar messages [ 3415.530133] Lustre: Skipped 5 previous similar messages [ 3425.247116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3425.302237] Lustre: server umount lustre-OST0000 complete [ 3429.343806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3432.664484] Lustre: server umount lustre-MDT0000 complete [ 3433.881218] LustreError: 178871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369871 with bad export cookie 9419704141061863571 [ 3433.883822] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3433.885710] LustreError: 178871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3434.001426] Lustre: server umount lustre-MDT0001 complete [ 3437.819283] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3438.920045] Key type lgssc unregistered [ 3439.070441] LNet: 182708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3439.072642] LNetError: 182708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3439.082338] LNet: Removed LNI 192.168.201.114@tcp [ 3439.398678] Key type .llcrypt unregistered [ 3439.400058] Key type ._llcrypt unregistered [ 3448.790135] Key type ._llcrypt registered [ 3448.790992] Key type .llcrypt registered [ 3448.833059] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3449.261384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3449.285916] alg: No test for adler32 (adler32-zlib) [ 3450.136091] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 3450.230392] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3451.807259] Key type lgssc registered [ 3452.205600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3454.856211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3454.862895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.999686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.298251] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3460.228365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.349086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3461.794509] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3464.339616] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3466.224503] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3468.550119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.663097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.592071] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3472.960772] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3481.954761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3496.416703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3496.419387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.425106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3497.442033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3497.442159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3497.444584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.251914] Lustre: server umount lustre-OST0000 complete [ 3506.656918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3506.659666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3506.664022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3506.666054] Lustre: Skipped 1 previous similar message [ 3508.756626] Lustre: server umount lustre-MDT0000 complete [ 3510.073333] LustreError: 186898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789369947 with bad export cookie 1581159463182763567 [ 3510.076490] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3510.077361] LustreError: 186898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3510.198580] Lustre: server umount lustre-MDT0001 complete [ 3513.836741] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 3514.939980] Key type lgssc unregistered [ 3515.070437] LNet: 187647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3515.073163] LNetError: 187647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3515.081314] LNet: Removed LNI 192.168.201.114@tcp [ 3515.398782] Key type .llcrypt unregistered [ 3515.400581] Key type ._llcrypt unregistered [ 3524.783640] Key type ._llcrypt registered [ 3524.784932] Key type .llcrypt registered [ 3524.835843] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3525.178595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3525.186678] alg: No test for adler32 (adler32-zlib) [ 3526.059204] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 3526.148984] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 3527.727142] Key type lgssc registered [ 3528.119311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3531.059068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3531.065182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.246132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.587455] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3536.570392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.705124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3538.092565] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3540.717342] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3542.641711] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3544.969513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3545.091492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.018261] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3549.381241] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3557.413067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3562.464699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3562.467143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.468021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3562.470776] Lustre: Skipped 1 previous similar message [ 3564.891736] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 3564.894591] Lustre: Skipped 1 previous similar message [ 3567.585984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3567.588027] Lustre: Skipped 1 previous similar message [ 3570.005117] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 3575.124506] Lustre: lustre-OST0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 3575.126736] Lustre: Skipped 2 previous similar messages [ 3576.287116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3576.338903] Lustre: server umount lustre-OST0000 complete [ 3578.748126] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9abd510b9000.ost_server_uuid 50 [ 3579.238574] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9abd510b9000.ost_server_uuid in DISCONN state after 0 sec [ 3579.758063] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3589.707425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.708023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3589.711659] Lustre: Skipped 1 previous similar message [ 3595.860223] Lustre: server umount lustre-MDT0000 complete [ 3597.057927] LustreError: 192041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370034 with bad export cookie 14022227597094502405 [ 3597.060258] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3597.061338] LustreError: 192041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3597.173892] Lustre: server umount lustre-MDT0001 complete [ 3599.866055] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:13:56 (1789370036) [ 3602.052487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.228953] LustreError: 192997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.235364] LustreError: 192997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3602.251466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.675966] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3606.484076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.631382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3607.983276] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3610.287611] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3611.973919] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3614.101381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3616.000193] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3618.169336] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3620.227127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.292871] Lustre: lustre-OST0001: new disk, initializing [ 3620.294487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3620.297398] Lustre: 195836:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3620.329942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.331515] Lustre: Skipped 1 previous similar message [ 3622.061226] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3625.556360] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3625.976459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3625.979698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3626.006921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3627.049982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3632.096441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3632.099848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.102494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3632.105176] Lustre: Skipped 1 previous similar message [ 3632.108907] Lustre: Skipped 4 previous similar messages [ 3637.757046] Lustre: server umount lustre-OST0000 complete [ 3639.795268] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3639.875353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3641.172593] LustreError: 196558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.240926] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3641.308882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3642.339230] LustreError: 194818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.343029] LustreError: 194818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3646.293163] LustreError: 195233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.579217] LustreError: 194818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.579258] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3652.586349] LustreError: 194818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3652.591389] Lustre: Skipped 1 previous similar message [ 3657.809868] Lustre: server umount lustre-OST0001 complete [ 3662.817750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.818081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3662.821801] Lustre: Skipped 1 previous similar message [ 3665.365601] Lustre: server umount lustre-MDT0000 complete [ 3666.503178] LustreError: 193024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370103 with bad export cookie 14022227597094503462 [ 3666.506859] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3666.507524] LustreError: 193024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3666.616651] Lustre: server umount lustre-MDT0001 complete [ 3673.859760] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:15:10 (1789370110) [ 3675.938184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.025065] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3676.037891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3676.100974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.358809] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3679.882211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3679.909471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3679.919541] Lustre: 199249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3681.211726] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3683.396508] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3685.045319] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3687.071693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3687.135606] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3689.139776] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3690.058621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3691.484109] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3693.561325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3693.585594] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3695.411140] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3697.711070] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3698.657711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3698.659065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3698.661315] Lustre: Skipped 1 previous similar message [ 3698.662811] Lustre: Skipped 9 previous similar messages [ 3704.568804] Lustre: server umount lustre-OST0000 complete [ 3704.802348] LustreError: 201097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.809920] LustreError: 201097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3720.163044] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.163293] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3721.229460] Lustre: server umount lustre-OST0001 complete [ 3728.537806] Lustre: server umount lustre-MDT0000 complete [ 3729.710107] LustreError: 199241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370167 with bad export cookie 14022227597094504848 [ 3729.712192] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.713192] LustreError: 199241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3729.828530] Lustre: server umount lustre-MDT0001 complete [ 3737.330683] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:16:14 (1789370174) [ 3742.043365] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 3744.732626] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 3755.582401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.669030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3755.679143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3755.712407] Lustre: lustre-MDT0000: new disk, initializing [ 3755.730436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.732529] Lustre: Skipped 3 previous similar messages [ 3755.736790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3757.023744] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3760.952429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.987474] Lustre: 207304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3761.007392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3761.008914] Lustre: Skipped 1 previous similar message [ 3761.042259] Lustre: lustre-MDT0001: new disk, initializing [ 3761.080899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3761.084136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3762.354286] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3765.966466] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3767.774264] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3769.881861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.992062] Lustre: lustre-OST0000: new disk, initializing [ 3769.993676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3769.996058] Lustre: 209271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3771.835627] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3775.246739] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3775.481991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3775.484973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3775.516269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3776.488243] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3776.555156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3777.767152] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3777.824145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3780.578191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.578595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.579742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.579746] Lustre: Skipped 8 previous similar messages [ 3780.579791] LustreError: Skipped 2 previous similar messages [ 3780.582424] Lustre: Skipped 4 previous similar messages [ 3784.525781] Lustre: server umount lustre-OST0000 complete [ 3800.031115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3800.096547] Lustre: server umount lustre-MDT0000 complete [ 3801.367258] LustreError: 208100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370238 with bad export cookie 14022227597094506367 [ 3801.369884] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3801.372116] LustreError: 208100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3801.487681] Lustre: server umount lustre-MDT0001 complete [ 3804.335660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3804.515172] LustreError: 211483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3804.521311] LustreError: 211483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3806.012756] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3809.036626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.418747] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3812.668829] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3814.385556] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3816.506635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3818.454893] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3820.816320] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3822.259376] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3829.456302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3830.939330] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3831.033140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3834.338529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3834.341482] LustreError: Skipped 1 previous similar message [ 3837.774355] Lustre: server umount lustre-OST0000 complete [ 3839.775415] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3841.217357] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3864.831458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.955666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.958507] Lustre: Skipped 5 previous similar messages [ 3866.918637] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3869.424825] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3869.921208] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3872.971312] Lustre: server umount lustre-OST0000 complete [ 3875.808194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3875.811042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.815313] Lustre: Skipped 4 previous similar messages [ 3876.315638] Lustre: server umount lustre-MDT0000 complete [ 3877.678242] LustreError: 211465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370314 with bad export cookie 14022227597094507235 [ 3877.682752] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3877.807543] Lustre: server umount lustre-MDT0001 complete [ 3880.603000] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:18:37 (1789370317) [ 3882.643035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.790727] LustreError: 217487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.077309] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3886.743665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.153339] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3890.519783] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3892.277162] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3894.424361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3896.314311] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3898.502474] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3899.853336] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3908.057666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3909.340321] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3909.415397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3911.505350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3911.577503] Lustre: lustre-OST0001: new disk, initializing [ 3911.579290] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3911.581990] Lustre: 220912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3913.322981] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3916.878432] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3918.169952] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3921.408100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3921.411803] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3921.443739] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3922.316960] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3923.718295] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3923.794564] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3926.498297] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.501776] LustreError: Skipped 1 previous similar message [ 3926.503509] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3926.506537] Lustre: Skipped 12 previous similar messages [ 3930.559168] Lustre: server umount lustre-OST0001 complete [ 3932.469488] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3932.540967] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3933.797797] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3933.865976] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3946.978714] LustreError: 219307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3946.982623] LustreError: 219307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3957.488832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3959.458711] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 3961.880466] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3962.403952] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3962.852543] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3962.856354] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3962.856498] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3962.863661] Lustre: Skipped 1 previous similar message [ 3965.115257] Lustre: server umount lustre-OST0001 complete [ 3983.244118] Lustre: server umount lustre-OST0000 complete [ 3990.548130] Lustre: server umount lustre-MDT0000 complete [ 3991.654022] LustreError: 219329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370428 with bad export cookie 14022227597094508166 [ 3991.657183] LustreError: 219329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3991.657285] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3997.817698] Lustre: server umount lustre-MDT0001 complete [ 4005.481109] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:20:42 (1789370442) [ 4007.724049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4007.819655] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4007.828904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4007.900587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.902754] Lustre: Skipped 5 previous similar messages [ 4009.256981] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4012.026012] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.060804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4012.071052] Lustre: 226173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4013.533455] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4015.900171] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4017.728660] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4019.938387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4020.006512] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4021.994901] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4024.295619] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4033.516377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4033.544602] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4035.546699] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4037.943914] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4038.463216] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4039.929394] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4040.007559] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4041.362933] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4041.436771] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4043.095290] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9abd48817800.ost_server_uuid 50 [ 4043.551213] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9abd48817800.ost_server_uuid in FULL state after 0 sec [ 4043.848457] Lustre: Permanently deactivating lustre-OST0001 [ 4043.853631] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4044.556434] Lustre: Permanently reactivating lustre-OST0001 [ 4049.891942] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.892699] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.897181] Lustre: Skipped 8 previous similar messages [ 4049.899279] LustreError: Skipped 3 previous similar messages [ 4051.511912] Lustre: server umount lustre-OST0001 complete [ 4058.769987] Lustre: server umount lustre-OST0000 complete [ 4066.067630] Lustre: server umount lustre-MDT0000 complete [ 4067.197917] LustreError: 226163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370504 with bad export cookie 14022227597094509440 [ 4067.201329] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4067.201450] LustreError: 226163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4067.302526] Lustre: server umount lustre-MDT0001 complete [ 4074.307768] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:21:51 (1789370511) [ 4076.599620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.699811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4076.708774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4076.711640] Lustre: Skipped 5 previous similar messages [ 4078.073212] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4080.876881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.910970] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4080.920338] Lustre: 232852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4082.333150] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4084.683090] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4086.577478] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4088.700836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4088.772390] Lustre: Permanently deactivating lustre-OST0000 [ 4088.776251] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4089.837064] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4090.538493] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4092.690767] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4094.710848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4096.585570] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4096.985985] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4098.943220] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4099.468342] Lustre: Permanently reactivating lustre-OST0000 [ 4106.747410] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4106.749957] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4106.755104] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4106.773201] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4106.780958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4116.729861] Lustre: server umount lustre-OST0001 complete [ 4117.476742] LustreError: 234698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.480678] LustreError: 234698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 4124.114111] Lustre: server umount lustre-OST0000 complete [ 4131.681747] Lustre: server umount lustre-MDT0000 complete [ 4132.905140] LustreError: 232844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370570 with bad export cookie 14022227597094511071 [ 4132.909001] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4132.909758] LustreError: 232844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4133.027716] Lustre: server umount lustre-MDT0001 complete [ 4136.543247] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4137.585878] Key type lgssc unregistered [ 4137.707373] LNet: 237724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4137.709559] LNetError: 237724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4137.716307] LNet: Removed LNI 192.168.201.114@tcp [ 4138.030311] Key type .llcrypt unregistered [ 4138.031479] Key type ._llcrypt unregistered [ 4146.294038] Key type ._llcrypt registered [ 4146.295043] Key type .llcrypt registered [ 4146.329119] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4146.638922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4146.651716] alg: No test for adler32 (adler32-zlib) [ 4147.489449] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 4147.570971] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4149.143129] Key type lgssc registered [ 4149.506986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4152.771849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4152.777197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.922669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.228207] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4157.929671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.965290] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4157.974669] Lustre: 239163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4157.978663] Lustre: Permanently deactivating lustre-MDT0001 [ 4157.983764] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4158.090587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4159.400229] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4159.852367] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4159.854381] LustreError: 239204:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4159.859687] LustreError: 239204:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4159.865550] LustreError: 239204:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 4161.775329] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4163.617699] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4165.893461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4166.005148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4167.984813] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4168.036704] LustreError: 241378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.040078] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4168.041975] LustreError: 241378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4170.593695] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4172.975103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4173.051207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4174.985715] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4177.359457] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4178.406831] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4179.432392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4179.432678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4183.304188] Lustre: Permanently reactivating lustre-MDT0001 [ 4183.310282] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4183.313027] Lustre: Skipped 2 previous similar messages [ 4191.067378] Lustre: lustre-MDT0001: Client ef5bcba7-17a6-4ff2-80f8-4086cdcfb637 (at 192.168.201.14@tcp) reconnecting [ 4193.404637] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4193.890515] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4195.645376] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4195.731120] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4197.178170] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4197.269730] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4198.083919] Lustre: Permanently deactivating lustre-MDT0001 [ 4198.089667] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4198.091626] Lustre: Skipped 2 previous similar messages [ 4205.061214] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4209.122405] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4209.125738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4209.131429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.144800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.145968] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.150353] Lustre: Skipped 1 previous similar message [ 4213.847954] Lustre: server umount lustre-MDT0000 complete [ 4215.185166] LustreError: 239199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370652 with bad export cookie 11021134203763185677 [ 4215.188716] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4215.190435] LustreError: 239199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4215.263647] LustreError: 239167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4215.269833] LustreError: 239167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4215.317606] Lustre: server umount lustre-MDT0001 complete [ 4222.818495] Lustre: server umount lustre-OST0000 complete [ 4230.051583] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:24:26 (1789370666) [ 4235.350312] Lustre: server umount lustre-OST0001 complete [ 4238.936930] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4244.635849] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4264.221439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.353930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4264.359583] Lustre: Skipped 2 previous similar messages [ 4264.371921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4264.416238] Lustre: lustre-MDT0000: new disk, initializing [ 4264.488782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.501046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4266.508362] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4271.902507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.947701] Lustre: 247418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4271.981055] LustreError: 247428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.985883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4271.989477] Lustre: Skipped 1 previous similar message [ 4272.021811] Lustre: lustre-MDT0001: new disk, initializing [ 4272.074213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4272.077429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4273.646042] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4278.235404] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4280.687068] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4283.540904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4283.677888] Lustre: lustre-OST0000: new disk, initializing [ 4283.680251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4283.683625] Lustre: 249385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4283.726681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.729573] Lustre: Skipped 1 previous similar message [ 4286.090676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4286.094661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4286.166410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4286.541072] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4290.682255] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4291.578901] LustreError: 247423:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4296.320203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4296.384245] LustreError: 249371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.398223] Lustre: lustre-OST0001: new disk, initializing [ 4296.400380] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4296.403289] Lustre: 250499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4298.789362] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4301.591058] LustreError: 247423:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4301.602709] LustreError: 250104:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4303.461053] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4304.155653] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4306.435657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4306.438942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4306.478474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4311.703122] LustreError: 250104:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4316.640362] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4316.643844] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.649216] Lustre: Skipped 1 previous similar message [ 4316.652869] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4316.655198] Lustre: Skipped 4 previous similar messages [ 4318.529548] Lustre: server umount lustre-OST0001 complete [ 4321.763162] LustreError: 249370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4321.769359] LustreError: 249370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4332.001932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.002137] LustreError: 249677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.003159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4332.003163] Lustre: Skipped 1 previous similar message [ 4332.007099] Lustre: Skipped 2 previous similar messages [ 4332.011412] LustreError: 249677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4337.122721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4337.124798] Lustre: Skipped 2 previous similar messages [ 4345.319738] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4345.449640] Lustre: server umount lustre-OST0000 complete [ 4348.896332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4348.905559] LustreError: Skipped 1 previous similar message [ 4348.910868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.923983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.926166] Lustre: Skipped 2 previous similar messages [ 4362.207289] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4362.333142] Lustre: server umount lustre-MDT0000 complete [ 4362.720917] LustreError: 247425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.741991] LustreError: 247425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4366.079059] LustreError: 247408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789370803 with bad export cookie 11021134203763188456 [ 4366.083518] LustreError: 247408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4366.089629] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.514251] Lustre: server umount lustre-MDT0001 complete [ 4383.329807] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4385.610583] Key type lgssc unregistered [ 4385.836763] LNet: 252531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4385.841278] LNetError: 252531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4385.852941] LNet: Removed LNI 192.168.201.114@tcp [ 4386.938355] Key type .llcrypt unregistered [ 4386.941545] Key type ._llcrypt unregistered [ 4424.287441] Key type ._llcrypt registered [ 4424.289288] Key type .llcrypt registered [ 4424.374799] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4425.479368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4425.502122] alg: No test for adler32 (adler32-zlib) [ 4426.497714] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 4426.736422] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4428.407372] Key type lgssc registered [ 4429.155736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4434.561432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4434.568700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.770359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4435.792678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4435.944214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.941539] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4444.970767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.036543] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4445.064477] Lustre: 254717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4445.255709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4447.941114] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4452.936604] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4457.235599] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4461.968491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4462.082848] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4462.210734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.095202] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4470.901498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4471.330064] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4517.345715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4517.353201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.356906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4517.359487] Lustre: Skipped 1 previous similar message [ 4517.367826] Lustre: Skipped 1 previous similar message [ 4522.465759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4522.471883] Lustre: Skipped 1 previous similar message [ 4523.150570] Lustre: server umount lustre-OST0000 complete [ 4526.340264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4535.000695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4535.284618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4539.108724] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4544.472427] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4547.578809] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.599626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.607491] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4547.621643] Lustre: Skipped 1 previous similar message [ 4583.394043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4583.405788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.406304] LustreError: Skipped 1 previous similar message [ 4583.406892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4583.420758] Lustre: Skipped 1 previous similar message [ 4587.875908] Lustre: server umount lustre-OST0000 complete [ 4593.633347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.638905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4593.643161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.643169] Lustre: Skipped 1 previous similar message [ 4593.655757] Lustre: Skipped 1 previous similar message [ 4597.229489] Lustre: server umount lustre-MDT0000 complete [ 4599.762913] LustreError: 254708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371037 with bad export cookie 14394359403859368558 [ 4599.772970] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.776993] LustreError: 254708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4600.069756] Lustre: server umount lustre-MDT0001 complete [ 4607.884955] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4610.074373] Key type lgssc unregistered [ 4610.344767] LNet: 260132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4610.361862] LNetError: 260132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4610.378740] LNet: Removed LNI 192.168.201.114@tcp [ 4611.033896] Key type .llcrypt unregistered [ 4611.035419] Key type ._llcrypt unregistered [ 4626.313314] Key type ._llcrypt registered [ 4626.315327] Key type .llcrypt registered [ 4626.372470] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4626.894406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4626.930413] alg: No test for adler32 (adler32-zlib) [ 4627.903089] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 4628.053430] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4629.679161] Key type lgssc registered [ 4630.464718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4635.386990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4635.442750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.737615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.312781] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4644.204096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.374661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4646.327532] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4650.854621] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4654.320350] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4658.908895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4659.161652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.258869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4663.058278] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4667.564924] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4686.987871] Lustre: server umount lustre-OST0000 complete [ 4689.888349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.889797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.893058] Lustre: Skipped 1 previous similar message [ 4695.011687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4695.014679] Lustre: Skipped 2 previous similar messages [ 4699.154709] Lustre: server umount lustre-MDT0000 complete [ 4700.129859] LustreError: 261520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.150627] LustreError: 261520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4701.374238] LustreError: 263361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371138 with bad export cookie 9847468492512536063 [ 4701.377307] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4701.380407] LustreError: 263361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4711.012471] Lustre: server umount lustre-MDT0001 complete [ 4717.451571] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4719.064237] Key type lgssc unregistered [ 4719.247625] LNet: 265820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4719.251213] LNetError: 265820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4719.260639] LNet: Removed LNI 192.168.201.114@tcp [ 4719.769619] Key type .llcrypt unregistered [ 4719.771303] Key type ._llcrypt unregistered [ 4730.538175] Key type ._llcrypt registered [ 4730.540531] Key type .llcrypt registered [ 4730.605479] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4731.215867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4731.236476] alg: No test for adler32 (adler32-zlib) [ 4732.157748] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 4732.290696] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4733.919230] Key type lgssc registered [ 4734.719826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4739.635066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4739.648632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.885157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4743.100306] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4748.924158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.261459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4752.047948] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4757.048143] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4760.900306] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4764.790578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4765.071534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.129513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4768.909664] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4772.726036] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4781.536083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4781.539776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.550102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4784.611902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4784.614087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4784.618067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4784.628077] Lustre: Skipped 1 previous similar message [ 4789.731705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4789.735978] Lustre: Skipped 1 previous similar message [ 4794.335119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4794.432127] Lustre: server umount lustre-OST0000 complete [ 4799.971230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.974473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4799.984359] Lustre: Skipped 1 previous similar message [ 4799.990639] Lustre: Skipped 1 previous similar message [ 4802.886577] Lustre: server umount lustre-MDT0000 complete [ 4804.913318] LustreError: 266997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371242 with bad export cookie 18345047974921215883 [ 4804.917898] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4804.923240] LustreError: 266997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4805.053779] Lustre: server umount lustre-MDT0001 complete [ 4811.355842] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4813.022184] Key type lgssc unregistered [ 4813.193572] LNet: 270688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4813.196816] LNetError: 270688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4813.212528] LNet: Removed LNI 192.168.201.114@tcp [ 4813.677314] Key type .llcrypt unregistered [ 4813.679124] Key type ._llcrypt unregistered [ 4825.804765] Key type ._llcrypt registered [ 4825.806367] Key type .llcrypt registered [ 4825.855402] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4826.281581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4826.290355] alg: No test for adler32 (adler32-zlib) [ 4827.175304] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 4827.296837] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4828.895203] Key type lgssc registered [ 4829.327466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4832.521394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4832.534800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.726689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.319873] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4839.824261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.026695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4841.850668] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4844.993684] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4847.117754] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4849.736318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4849.870963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.071493] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4854.957840] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4859.110768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4871.648445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4871.652368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4871.659113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4872.133742] Lustre: server umount lustre-OST0000 complete [ 4874.721065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.722219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4874.725345] Lustre: Skipped 1 previous similar message [ 4879.840342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4879.843227] Lustre: Skipped 2 previous similar messages [ 4883.961139] Lustre: server umount lustre-MDT0000 complete [ 4884.960097] LustreError: 272077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.967307] LustreError: 272077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4885.395572] LustreError: 272058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371322 with bad export cookie 9469209868439617171 [ 4885.399905] LustreError: 272058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4885.400024] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4885.527717] Lustre: server umount lustre-MDT0001 complete [ 4889.436879] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4890.606701] Key type lgssc unregistered [ 4890.743458] LNet: 276377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4890.746391] LNetError: 276377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4890.758320] LNet: Removed LNI 192.168.201.114@tcp [ 4891.099348] Key type .llcrypt unregistered [ 4891.100646] Key type ._llcrypt unregistered [ 4898.708394] Key type ._llcrypt registered [ 4898.710209] Key type .llcrypt registered [ 4898.759404] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4899.086240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4899.101837] alg: No test for adler32 (adler32-zlib) [ 4899.997693] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 4900.087506] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4901.679111] Key type lgssc registered [ 4902.097388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4905.217705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4905.227115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.444250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4908.126338] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4911.745398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.886222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4913.636226] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4916.728425] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4919.132279] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4922.189690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4922.388878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4924.914596] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 4927.833147] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4931.621659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4944.865127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4944.867843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.872191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4946.913605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4946.914243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4946.920629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.308498] Lustre: server umount lustre-OST0000 complete [ 4952.033863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.034700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4952.039339] Lustre: Skipped 1 previous similar message [ 4952.043313] Lustre: Skipped 1 previous similar message [ 4957.154829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4957.156812] Lustre: Skipped 1 previous similar message [ 4962.273600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4962.279118] Lustre: Skipped 1 previous similar message [ 4964.319113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4964.387938] Lustre: server umount lustre-MDT0000 complete [ 4965.847511] LustreError: 277554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371403 with bad export cookie 7717320957824543824 [ 4965.848900] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4965.854710] LustreError: 277554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4965.996205] Lustre: server umount lustre-MDT0001 complete [ 4970.118892] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 4971.293973] Key type lgssc unregistered [ 4971.439595] LNet: 281302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4971.442262] LNetError: 281302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4971.450322] LNet: Removed LNI 192.168.201.114@tcp [ 4971.814301] Key type .llcrypt unregistered [ 4971.815833] Key type ._llcrypt unregistered [ 4975.920476] Key type ._llcrypt registered [ 4975.921920] Key type .llcrypt registered [ 4975.978465] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4982.874874] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 4993.848475] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 4994.747926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4994.963800] alg: No test for adler32 (adler32-zlib) [ 4996.107221] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 4996.284347] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 4998.199153] Key type lgssc registered [ 5001.946589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5068.753490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5068.879887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.380872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5070.442454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5070.643565] Lustre: lustre-MDT0000: new disk, initializing [ 5070.846901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.896573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5080.463702] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5092.599441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.733299] Lustre: 284831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5092.813826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5092.822438] Lustre: Skipped 1 previous similar message [ 5092.923652] Lustre: lustre-MDT0001: new disk, initializing [ 5093.057781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5093.129029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5093.166047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5098.321792] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5105.808182] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5111.379892] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5118.075898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5118.278171] Lustre: lustre-OST0000: new disk, initializing [ 5118.281658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5118.287348] Lustre: 286801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5118.364786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.981382] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5126.735172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5126.749793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5126.871173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5132.362758] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5136.420946] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5136.613861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5140.490660] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5140.755648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5147.623439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5147.627240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.632664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5147.634802] LustreError: Skipped 1 previous similar message [ 5147.650966] Lustre: Skipped 1 previous similar message [ 5148.833631] Lustre: server umount lustre-OST0000 complete [ 5152.738171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.751393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.753851] Lustre: Skipped 1 previous similar message [ 5157.860136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5157.862610] Lustre: Skipped 1 previous similar message [ 5158.646866] Lustre: server umount lustre-MDT0000 complete [ 5162.156175] LustreError: 286800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371599 with bad export cookie 321946989190114662 [ 5162.168143] LustreError: 286800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5162.169428] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.695954] Lustre: server umount lustre-MDT0001 complete [ 5170.286384] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:40:05 (1789371605) [ 5176.323584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.777284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.667552] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5181.934895] LustreError: 289213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5181.963130] LustreError: 289213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5187.052055] LustreError: 289212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.258373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.585489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5195.127802] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5202.006427] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5207.215130] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5213.228801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5213.542853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5219.631782] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5226.402725] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5236.201850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5236.213941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.232023] Lustre: Skipped 1 previous similar message [ 5236.235752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5237.216875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5240.341352] Lustre: server umount lustre-OST0000 complete [ 5246.944445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5246.955211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5246.967258] Lustre: Skipped 1 previous similar message [ 5246.973954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5246.983763] Lustre: Skipped 2 previous similar messages [ 5257.698617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.711732] Lustre: Skipped 5 previous similar messages [ 5258.209826] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5258.321416] Lustre: server umount lustre-MDT0000 complete [ 5261.503719] LustreError: 289193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371698 with bad export cookie 321946989190115544 [ 5261.511071] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.513321] LustreError: 289193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5261.696420] Lustre: server umount lustre-MDT0001 complete [ 5271.295742] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5275.041408] Key type lgssc unregistered [ 5275.371105] LNet: 292748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5275.384973] LNetError: 292748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5275.404348] LNet: Removed LNI 192.168.201.114@tcp [ 5277.811666] Key type .llcrypt unregistered [ 5277.813390] Key type ._llcrypt unregistered [ 5312.401905] Key type ._llcrypt registered [ 5312.403880] Key type .llcrypt registered [ 5312.532405] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5327.642443] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5328.745496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5328.922303] alg: No test for adler32 (adler32-zlib) [ 5330.119805] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 5330.369948] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5332.039568] Key type lgssc registered [ 5332.975869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5379.747713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5379.795347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5381.065322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5381.096509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5381.209560] Lustre: lustre-MDT0000: new disk, initializing [ 5381.300504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5381.320948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5385.530953] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5397.085491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.285357] Lustre: 296573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5397.347710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5397.351362] Lustre: Skipped 1 previous similar message [ 5397.447314] Lustre: lustre-MDT0001: new disk, initializing [ 5397.565397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5397.611933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5397.622677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5403.878979] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5414.413453] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5420.355651] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5427.756621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.052710] Lustre: lustre-OST2710: new disk, initializing [ 5428.059938] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5428.066626] Lustre: 298554:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5428.174685] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5434.654831] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5436.988089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5436.994836] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5437.079013] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5444.498945] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5451.696362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5451.874968] Lustre: lustre-OST03e8: new disk, initializing [ 5451.879677] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5451.889986] Lustre: 299616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5451.984046] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5457.980895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5457.992157] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5458.127914] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5458.225598] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5464.770416] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5471.268996] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5471.502227] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5476.185433] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5476.380917] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5480.925587] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5481.135612] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5485.660940] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5485.852336] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5493.006552] Lustre: server umount lustre-MDT0000 complete [ 5494.252911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.276267] Lustre: Skipped 1 previous similar message [ 5497.168604] LustreError: 296566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789371934 with bad export cookie 2231154731604050133 [ 5497.173460] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.187472] LustreError: 296566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5497.520688] Lustre: server umount lustre-MDT0001 complete [ 5512.311361] Lustre: server umount lustre-OST2710 complete [ 5515.235083] Lustre: 294371:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789371936/real 1789371936] req@00000000e7f08d04 x1876292290021376/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789371952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5515.275230] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.304099] Lustre: Skipped 2 previous similar messages [ 5516.615501] Lustre: server umount lustre-OST03e8 complete [ 5536.534591] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 5548.318881] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5586.824110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.051981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5587.076675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5587.136424] Lustre: lustre-MDT0000: new disk, initializing [ 5587.223980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5587.259399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5593.086566] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5603.920307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.059492] Lustre: 304866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5604.111750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5604.118602] Lustre: Skipped 1 previous similar message [ 5604.176571] Lustre: lustre-MDT0001: new disk, initializing [ 5604.278213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5604.317557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5604.339918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5608.534145] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5617.718865] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5623.592482] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5630.771202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5631.108312] Lustre: lustre-OST0000: new disk, initializing [ 5631.113982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5631.119337] Lustre: 306833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5631.224511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.683838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5637.696944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5637.814812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5638.781700] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5648.292494] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5653.169738] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5653.372738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5658.571839] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5658.901705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5663.714552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.716750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5663.748491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5663.767912] Lustre: Skipped 1 previous similar message [ 5668.838922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5668.851786] Lustre: Skipped 1 previous similar message [ 5673.958535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.968147] Lustre: Skipped 1 previous similar message [ 5674.975254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5675.193531] Lustre: server umount lustre-OST0000 complete [ 5681.120942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5681.125974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.138876] Lustre: Skipped 1 previous similar message [ 5681.143557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5686.637497] Lustre: server umount lustre-MDT0000 complete [ 5689.312583] LustreError: 304873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5689.329471] LustreError: 304873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5690.540700] LustreError: 304858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372127 with bad export cookie 2231154731604051813 [ 5690.550685] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5690.552381] LustreError: 304858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5690.986890] Lustre: server umount lustre-MDT0001 complete [ 5700.304961] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:48:55 (1789372135) [ 5702.000980] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5703.662533] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:48:59 (1789372139) [ 5707.718757] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5735.212812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5735.445837] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5735.470959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5735.718794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.282350] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5753.198755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.259346] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5753.288548] Lustre: 310691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5761.508701] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5789.117382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5789.420638] Lustre: Denying initial registration attempt from nid 192.168.201.114@tcp, specified as failover [ 5789.428351] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5789.433785] LustreError: 312149:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 5789.449212] LustreError: 312149:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5789.453603] LustreError: 312149:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5789.565328] Lustre: server umount lustre-OST0000 complete [ 5789.576665] LustreError: 312149:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5797.861768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.878869] Lustre: Skipped 1 previous similar message [ 5797.884127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5797.895105] Lustre: Skipped 3 previous similar messages [ 5802.824767] Lustre: server umount lustre-MDT0000 complete [ 5802.977055] LustreError: 311704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.069928] LustreError: 310686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372244 with bad export cookie 2231154731604052695 [ 5807.083188] LustreError: 310686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5807.084882] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5807.472742] Lustre: server umount lustre-MDT0001 complete [ 5820.710452] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 5823.909335] Key type lgssc unregistered [ 5824.303894] LNet: 313270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5824.313762] LNetError: 313270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5824.329069] LNet: Removed LNI 192.168.201.114@tcp [ 5825.297444] Key type .llcrypt unregistered [ 5825.299715] Key type ._llcrypt unregistered [ 5838.575910] Key type ._llcrypt registered [ 5838.579822] Key type .llcrypt registered [ 5838.719764] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5840.301015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5840.320323] alg: No test for adler32 (adler32-zlib) [ 5841.488793] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 5841.907434] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 5843.552293] Key type lgssc registered [ 5845.221166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5895.568680] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 5906.001317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5906.049385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.542320] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5907.615262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5907.850983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.254766] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5922.787390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.961327] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5923.013724] Lustre: 316032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5923.552375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5929.256188] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5937.349671] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5943.524907] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5964.598682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.900710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5965.123253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5972.441776] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 5979.650822] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5984.224460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5984.229796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.244951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5984.740538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5984.754044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.772901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5987.924137] Lustre: server umount lustre-OST0000 complete [ 5994.979354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5994.982220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.983921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5995.007803] Lustre: Skipped 1 previous similar message [ 5995.018586] Lustre: Skipped 1 previous similar message [ 5998.918583] Lustre: server umount lustre-MDT0000 complete [ 6002.563973] LustreError: 317896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372439 with bad export cookie 18404298790056819087 [ 6002.574823] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6002.579475] LustreError: 317896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6002.979418] Lustre: server umount lustre-MDT0001 complete [ 6013.414607] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6016.071957] Key type lgssc unregistered [ 6016.416862] LNet: 319603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6016.423995] LNetError: 319603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6016.443852] LNet: Removed LNI 192.168.201.114@tcp [ 6017.344773] Key type .llcrypt unregistered [ 6017.347232] Key type ._llcrypt unregistered [ 6041.597644] Key type ._llcrypt registered [ 6041.600730] Key type .llcrypt registered [ 6041.710889] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6042.428931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6042.545218] alg: No test for adler32 (adler32-zlib) [ 6043.625736] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 6043.851379] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6045.495193] Key type lgssc registered [ 6046.872425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6055.934986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6055.990524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.513744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.048955] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6067.680787] LustreError: 321016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.701917] LustreError: 321016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6072.250572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6072.738879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6077.360257] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6084.893438] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6089.835204] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6100.075121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6100.765433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.317474] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6116.624295] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6124.000239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6124.018289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6124.033489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6127.607707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6127.610884] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.625389] Lustre: Skipped 1 previous similar message [ 6127.656077] Lustre: Skipped 1 previous similar message [ 6130.182881] Lustre: server umount lustre-MDT0000 complete [ 6132.706306] LustreError: 321015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.734205] LustreError: 321015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6134.113975] LustreError: 320998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372571 with bad export cookie 10657796272793075138 [ 6134.121687] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6134.130250] LustreError: 320998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6137.825513] LustreError: 321010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.829689] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.838291] LustreError: 321010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6137.876962] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6137.880804] Lustre: Skipped 2 previous similar messages [ 6139.872730] LustreError: 323507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6139.909059] LustreError: 323507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6140.673047] Lustre: server umount lustre-MDT0001 complete [ 6146.053328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6155.515038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.330209] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x93e8193e20505c15 [ 6159.344426] Lustre: MGC192.168.201.114@tcp: Connection restored to 0@lo (at 0@lo) [ 6159.659671] LustreError: 322846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.742706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6159.797746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6163.713347] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6168.930666] LustreError: 322847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.942991] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6168.956134] LustreError: 322847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6172.035744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.596550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6177.786132] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6178.400372] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6186.182988] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6188.021756] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6193.411727] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6195.045748] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6200.802157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6200.808108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.820076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6200.824470] Lustre: Skipped 2 previous similar messages [ 6203.370817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6206.441462] Lustre: server umount lustre-OST0000 complete [ 6213.603522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6213.603747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6213.619137] Lustre: Skipped 1 previous similar message [ 6213.620541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6213.620548] Lustre: Skipped 2 previous similar messages [ 6217.271488] Lustre: server umount lustre-MDT0000 complete [ 6218.720782] LustreError: 324611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.733255] LustreError: 324611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6221.254669] LustreError: 320996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372658 with bad export cookie 10657796272793082901 [ 6221.268829] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6221.587395] Lustre: server umount lustre-MDT0001 complete [ 6232.532972] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6235.273408] Key type lgssc unregistered [ 6235.547784] LNet: 327283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6235.555352] LNetError: 327283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6235.569363] LNet: Removed LNI 192.168.201.114@tcp [ 6236.389329] Key type .llcrypt unregistered [ 6236.393562] Key type ._llcrypt unregistered [ 6262.155421] Key type ._llcrypt registered [ 6262.157609] Key type .llcrypt registered [ 6262.284310] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6263.673663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6263.712554] alg: No test for adler32 (adler32-zlib) [ 6264.811901] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 6265.141801] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6266.791143] Key type lgssc registered [ 6267.964920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6276.724939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6276.743483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6278.139365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6283.219606] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6288.352581] LustreError: 328698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.374591] LustreError: 328698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6291.879374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.246391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6298.078813] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6305.388355] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6311.400907] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6316.770070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.788969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6316.795567] Lustre: Skipped 1 previous similar message [ 6316.813187] Lustre: Skipped 1 previous similar message [ 6322.148769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6322.159049] Lustre: Skipped 1 previous similar message [ 6327.265573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.314749] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6329.525885] Lustre: server umount lustre-MDT0000 complete [ 6332.388806] LustreError: 329426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.401438] LustreError: 329426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6333.550347] LustreError: 328678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372770 with bad export cookie 12102479881135185997 [ 6333.558464] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.570754] LustreError: 328678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6334.030466] Lustre: server umount lustre-MDT0001 complete [ 6348.810483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6349.184282] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6349.224067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6349.507288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6354.315603] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6363.624570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.728963] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6363.750308] Lustre: 331287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6363.958587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6368.801211] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6377.076833] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6383.262710] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6393.267284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6393.634870] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6393.651303] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6393.666418] LustreError: 333170:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 6393.673302] LustreError: 333170:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6393.688108] LustreError: 333170:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6393.841662] Lustre: server umount lustre-OST0000 complete [ 6393.846233] LustreError: 333170:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6405.308103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6405.566782] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6405.847061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.924949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6413.242196] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6421.396199] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6430.176695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6430.306263] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6430.397755] Lustre: lustre-OST0001: new disk, initializing [ 6430.402499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6430.408461] Lustre: 334725:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6430.513970] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6436.978147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6436.988307] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6437.138037] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6437.568192] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6446.470660] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6452.713746] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.715436] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6452.720859] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6452.732105] Lustre: Skipped 1 previous similar message [ 6452.739803] Lustre: Skipped 2 previous similar messages [ 6455.740325] Lustre: server umount lustre-OST0001 complete [ 6457.832381] LustreError: 335530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.852573] LustreError: 335530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6462.962395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6462.967665] LustreError: 333638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.971977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.975029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6462.976723] LustreError: Skipped 1 previous similar message [ 6463.044900] LustreError: 333638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6465.666925] Lustre: server umount lustre-OST0000 complete [ 6471.648177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6471.657109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.678455] Lustre: Skipped 1 previous similar message [ 6471.685879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6471.694605] Lustre: Skipped 1 previous similar message [ 6475.469759] Lustre: server umount lustre-MDT0000 complete [ 6478.308996] LustreError: 332106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.324136] LustreError: 332106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6479.446513] LustreError: 331280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789372916 with bad export cookie 12102479881135186550 [ 6479.450540] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6479.461311] LustreError: 331280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6480.168429] Lustre: server umount lustre-MDT0001 complete [ 6491.047108] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 6494.418621] Key type lgssc unregistered [ 6494.889644] LNet: 336682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6494.899120] LNetError: 336682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6495.975790] LNet: Removed LNI 192.168.201.114@tcp [ 6497.122163] Key type .llcrypt unregistered [ 6497.123833] Key type ._llcrypt unregistered [ 6573.685615] Key type ._llcrypt registered [ 6573.695462] Key type .llcrypt registered [ 6573.780098] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 6589.073144] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 6590.606493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6590.643777] alg: No test for adler32 (adler32-zlib) [ 6591.961293] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 6592.286988] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 6593.959249] Key type lgssc registered [ 6595.202747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6623.849593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6623.873491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.182864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6625.218511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6625.284428] Lustre: lustre-MDT0000: new disk, initializing [ 6625.393097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6625.407307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6629.208483] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6638.021533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.115845] Lustre: 341618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6638.163537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6638.167298] Lustre: Skipped 1 previous similar message [ 6638.227861] Lustre: lustre-MDT0001: new disk, initializing [ 6638.325797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.378043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6638.387173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6642.801784] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6650.868203] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6655.937853] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6661.567425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.794319] Lustre: lustre-OST0000: new disk, initializing [ 6661.797412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6661.802916] Lustre: 343586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6661.897855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6667.857494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6667.876638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6668.041473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6668.049788] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6676.862141] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6681.318543] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6681.507777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6685.591928] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6685.755525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6688.738278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6688.742873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.752348] LustreError: Skipped 1 previous similar message [ 6688.753482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6688.791688] Lustre: Skipped 1 previous similar message [ 6693.746243] Lustre: server umount lustre-OST0000 complete [ 6698.981773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6698.984674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6698.989167] Lustre: Skipped 1 previous similar message [ 6699.001233] Lustre: Skipped 2 previous similar messages [ 6703.851765] Lustre: server umount lustre-MDT0000 complete [ 6707.373777] LustreError: 342410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373144 with bad export cookie 16601356391597129693 [ 6707.375942] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6707.384148] LustreError: 342410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6707.737745] Lustre: server umount lustre-MDT0001 complete [ 6716.302746] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:05:51 (1789373151) [ 6724.354794] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:05:59 (1789373159) [ 6732.327715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.804758] LustreError: 346338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6732.822427] LustreError: 346338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6732.865845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.000452] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6737.903456] LustreError: 346339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.008586] LustreError: 346338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.749433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.105235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6749.050683] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6754.780944] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.253096] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6764.211596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6764.429439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.655480] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6776.380307] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6784.758443] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6786.695464] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6788.278215] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6790.000124] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6792.209198] Lustre: Failing over lustre-MDT0000 [ 6792.561353] Lustre: server umount lustre-MDT0000 complete [ 6793.185788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6793.191546] LustreError: 346339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.196537] Lustre: Skipped 1 previous similar message [ 6798.306826] LustreError: 346338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6798.323518] LustreError: 346338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6808.554440] LustreError: 346339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.574663] LustreError: 346339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6809.573835] Lustre: 339840:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789373230/real 1789373230] req@00000000ca7e368a x1876293613323776/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789373246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6809.592395] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6810.365078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.784644] LustreError: 339836:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e8d30db3 x1876293613329280/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6818.963097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.985244] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6820.184631] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6822.410404] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6824.438724] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6824.481108] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6824.531915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6832.028772] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6833.809499] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6838.123196] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6840.757760] Lustre: Failing over lustre-MDT0000 [ 6841.252864] Lustre: server umount lustre-MDT0000 complete [ 6844.898203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.902805] LustreError: 347066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.916025] Lustre: Skipped 2 previous similar messages [ 6844.974081] LustreError: 347066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6860.971499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.155517] Lustre: 339838:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789373282/real 1789373282] req@00000000f0c6ecb3 x1876293613345280/t0(0) o400->MGC192.168.201.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789373298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6861.224639] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6871.526856] LustreError: 339836:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f0c6ecb3 x1876293613351424/t0(0) o250->MGC192.168.201.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6871.893537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6871.935548] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6875.437572] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6876.502695] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6877.165619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6877.176942] Lustre: Skipped 2 previous similar messages [ 6877.212190] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6877.252140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6885.317280] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6887.884460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6902.752618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6902.761222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6902.776846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6902.797454] Lustre: Skipped 3 previous similar messages [ 6902.804734] Lustre: Skipped 1 previous similar message [ 6904.404558] Lustre: server umount lustre-MDT0000 complete [ 6907.872591] LustreError: 347066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.921734] LustreError: 347066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 6908.977151] LustreError: 346319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373346 with bad export cookie 16601356391597132409 [ 6908.983244] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.995694] LustreError: 346319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6909.330190] Lustre: server umount lustre-MDT0001 complete [ 6924.367925] Lustre: server umount lustre-OST0000 complete [ 6938.030805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6938.568605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.495575] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6951.637091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6957.532325] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6964.676169] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6971.438089] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6982.486545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6984.352474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6991.670235] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 6999.644123] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7005.152652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7005.158920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7005.176138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.241529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7010.882350] Lustre: server umount lustre-OST0000 complete [ 7018.479981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7018.480911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7018.485961] Lustre: Skipped 2 previous similar messages [ 7021.971527] Lustre: server umount lustre-MDT0000 complete [ 7023.587348] LustreError: 352684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.620396] LustreError: 352684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7026.015386] LustreError: 352664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373463 with bad export cookie 16601356391597133144 [ 7026.025996] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7026.027487] LustreError: 352664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7026.303749] Lustre: server umount lustre-MDT0001 complete [ 7038.494190] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7041.489969] Key type lgssc unregistered [ 7041.794838] LNet: 356216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7041.803775] LNetError: 356216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7041.840263] LNet: Removed LNI 192.168.201.114@tcp [ 7042.622836] Key type .llcrypt unregistered [ 7042.626275] Key type ._llcrypt unregistered [ 7071.066954] Key type ._llcrypt registered [ 7071.076595] Key type .llcrypt registered [ 7071.249141] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7073.014733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7073.068562] alg: No test for adler32 (adler32-zlib) [ 7074.270949] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 7074.480923] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7076.215398] Key type lgssc registered [ 7077.683811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7086.700206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7086.745909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.302938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.036976] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7097.826475] LustreError: 357729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.859177] LustreError: 357729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7102.663800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7102.970552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7103.242467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7108.761626] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7116.976951] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7123.333734] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7131.363247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.741753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.820852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7138.740284] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7147.520189] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7159.781573] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7159.783395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7159.786619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7159.801241] Lustre: Skipped 1 previous similar message [ 7160.713342] Lustre: server umount lustre-MDT0000 complete [ 7164.901617] LustreError: 357729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7164.918143] LustreError: 357729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7164.941904] LustreError: 357710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373602 with bad export cookie 8390740867612834879 [ 7164.969352] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7164.973221] LustreError: 357710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7165.795596] Lustre: server umount lustre-MDT0001 complete [ 7180.660179] Lustre: server umount lustre-OST0000 complete [ 7195.711768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.160253] LustreError: 361442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.231807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7201.252358] LustreError: 361441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.497181] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7206.376056] LustreError: 361441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7209.870201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.202265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7215.962672] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7223.584369] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7229.529549] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7236.891558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.417423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7241.519296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7244.726434] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7253.202292] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7266.273163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.281181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7266.291807] Lustre: Skipped 3 previous similar messages [ 7266.301332] Lustre: Skipped 4 previous similar messages [ 7271.394409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.815035] Lustre: server umount lustre-MDT0000 complete [ 7276.332646] LustreError: 361422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373713 with bad export cookie 8390740867612835992 [ 7276.353342] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7276.514936] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7276.517707] LustreError: 361438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.552969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7276.593550] Lustre: Skipped 2 previous similar messages [ 7276.828922] Lustre: server umount lustre-MDT0001 complete [ 7284.061445] Lustre: server umount lustre-OST0000 complete [ 7300.994375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.361965] LustreError: 365354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.525447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.502978] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7315.922144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.287316] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7328.978805] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7335.449892] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7343.329676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7343.874913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7343.880852] Lustre: Skipped 1 previous similar message [ 7345.984183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7350.860605] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7358.657659] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7366.626401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7366.636383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.644829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7366.676575] Lustre: Skipped 1 previous similar message [ 7366.694359] Lustre: Skipped 1 previous similar message [ 7369.788700] Lustre: server umount lustre-OST0000 complete [ 7376.867252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.885688] Lustre: Skipped 1 previous similar message [ 7380.340699] Lustre: server umount lustre-MDT0000 complete [ 7381.986957] LustreError: 365355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.997334] LustreError: 365355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7383.555918] LustreError: 367196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789373820 with bad export cookie 8390740867612837014 [ 7383.566505] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7383.573952] LustreError: 367196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7383.843289] Lustre: server umount lustre-MDT0001 complete [ 7396.760555] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7400.092958] Key type lgssc unregistered [ 7400.566572] LNet: 368889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7400.578483] LNetError: 368889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7401.641679] LNet: Removed LNI 192.168.201.114@tcp [ 7403.003472] Key type .llcrypt unregistered [ 7403.006501] Key type ._llcrypt unregistered [ 7430.994013] Key type ._llcrypt registered [ 7430.997941] Key type .llcrypt registered [ 7431.155277] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7432.230608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7432.330588] alg: No test for adler32 (adler32-zlib) [ 7433.353450] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 7433.620789] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7435.287891] Key type lgssc registered [ 7436.432269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7447.227460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7447.280595] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.301178] LustreError: 370329:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7447.357701] LustreError: 370329:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7447.367069] LustreError: 370329:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7447.379746] LustreError: 370329:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7447.396737] LustreError: 370329:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7458.163962] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7458.170516] LustreError: 370727:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7458.181879] LustreError: 370727:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7458.185940] LustreError: 370727:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7458.190327] LustreError: 370727:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7458.194663] LustreError: 370727:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7484.736883] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7488.025969] Key type lgssc unregistered [ 7488.309558] LNet: 371731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7488.324552] LNetError: 371731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.355161] LNet: Removed LNI 192.168.201.114@tcp [ 7489.688535] Key type .llcrypt unregistered [ 7489.691310] Key type ._llcrypt unregistered [ 7509.547866] Key type ._llcrypt registered [ 7509.550971] Key type .llcrypt registered [ 7509.673035] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 7526.700224] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7528.293388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7528.308982] alg: No test for adler32 (adler32-zlib) [ 7529.384794] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 7529.642933] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7531.335490] Key type lgssc registered [ 7532.698189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7573.618941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7573.645254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.893693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7574.926063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7575.009975] Lustre: lustre-MDT0000: new disk, initializing [ 7575.171537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.210292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7579.983217] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7589.797657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.870620] Lustre: 374884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7589.899370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7589.901865] Lustre: Skipped 1 previous similar message [ 7589.940760] Lustre: lustre-MDT0001: new disk, initializing [ 7590.039717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7590.126682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7590.147343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7596.500161] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7606.856592] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7613.791142] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7621.239971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7621.616147] Lustre: lustre-OST0000: new disk, initializing [ 7621.619885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7621.628705] Lustre: 376854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7621.728421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.058385] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7628.375565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7628.384422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7628.485843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7637.517823] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7642.493470] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7642.704997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7647.409490] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7647.719524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7654.368839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7654.378558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.390395] LustreError: Skipped 1 previous similar message [ 7654.393102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7654.402257] Lustre: Skipped 1 previous similar message [ 7654.419895] Lustre: Skipped 1 previous similar message [ 7656.290378] Lustre: server umount lustre-OST0000 complete [ 7659.488366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7659.499245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7659.514623] Lustre: Skipped 1 previous similar message [ 7659.523399] Lustre: Skipped 1 previous similar message [ 7664.608721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7665.958720] Lustre: server umount lustre-MDT0000 complete [ 7669.440065] LustreError: 374878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374106 with bad export cookie 7722803717053349397 [ 7669.441992] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7669.455068] LustreError: 374878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7669.896929] Lustre: server umount lustre-MDT0001 complete [ 7682.087986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7685.109197] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:21:59 (1789374119) [ 7694.202582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.842382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7699.943443] LustreError: 379316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.983483] LustreError: 379316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7700.672907] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7705.063102] LustreError: 379315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.950729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.230974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7713.498983] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7721.390281] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7727.762893] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7734.679746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7735.115357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.261624] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7751.598428] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7761.718907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7762.000479] Lustre: lustre-OST0001: new disk, initializing [ 7762.004068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7762.011405] Lustre: 382152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7762.146778] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7767.717256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7767.743878] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7767.945792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7771.382579] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7781.312808] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7788.519174] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7788.524357] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.531097] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7788.531106] Lustre: Skipped 1 previous similar message [ 7788.534993] LustreError: Skipped 1 previous similar message [ 7788.545760] Lustre: Skipped 1 previous similar message [ 7792.342361] Lustre: server umount lustre-OST0001 complete [ 7793.665834] LustreError: 382162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7793.680374] LustreError: 382162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7798.759859] LustreError: 381135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7798.773329] LustreError: 381135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7803.877123] LustreError: 381136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.896918] LustreError: 381136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7812.067936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7812.067980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.076064] LustreError: Skipped 1 previous similar message [ 7812.077242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7812.077247] Lustre: Skipped 2 previous similar messages [ 7812.121488] Lustre: Skipped 1 previous similar message [ 7814.142245] LustreError: 382162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7814.166244] LustreError: 382162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7815.731600] Lustre: server umount lustre-OST0000 complete [ 7821.797428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7821.808494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.826066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7821.830147] Lustre: Skipped 3 previous similar messages [ 7827.171018] Lustre: server umount lustre-MDT0000 complete [ 7830.654412] LustreError: 379295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374267 with bad export cookie 7722803717053350279 [ 7830.674922] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7837.295933] Lustre: server umount lustre-MDT0001 complete [ 7850.280398] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 7853.426566] Key type lgssc unregistered [ 7853.874340] LNet: 384144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7853.886396] LNetError: 384144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7853.910521] LNet: Removed LNI 192.168.201.114@tcp [ 7854.728713] Key type .llcrypt unregistered [ 7854.730450] Key type ._llcrypt unregistered [ 7917.552487] Key type ._llcrypt registered [ 7917.554507] Key type .llcrypt registered [ 7917.626430] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 7919.485239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7919.565953] alg: No test for adler32 (adler32-zlib) [ 7920.777265] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 7921.156118] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 7922.911682] Key type lgssc registered [ 7924.110443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7932.821797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7932.862417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.296697] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7934.357475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7934.753628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7940.061723] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7948.859974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.904676] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7948.915845] Lustre: 386940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7949.040092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7953.548782] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 7961.161466] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7966.056623] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7969.763922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7969.767517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7969.773242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7971.301624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7971.303043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.502734] Lustre: server umount lustre-MDT0000 complete [ 7977.660705] LustreError: 386934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374414 with bad export cookie 8458238613430662272 [ 7977.665103] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7977.670425] LustreError: 386934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7977.962854] Lustre: server umount lustre-MDT0001 complete [ 7988.435697] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:27:03 (1789374423) [ 7996.187732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.592044] LustreError: 389453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.611221] LustreError: 389453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7996.664148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8001.252548] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8002.019189] LustreError: 389454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8007.141928] LustreError: 389453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8009.463420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.979138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8014.063481] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8021.232598] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8027.425902] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8034.898649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8035.147598] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8035.393668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8042.316133] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8050.061869] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8057.409702] Lustre: Permanently deactivating lustre-OST0000 [ 8057.424389] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8065.558613] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8067.601553] LustreError: 392441:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8067.608034] LustreError: 392441:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8067.614806] LustreError: 392441:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8070.798120] LustreError: 392489:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8070.801433] LustreError: 392489:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8079.120125] Lustre: server umount lustre-OST0000 complete [ 8082.376785] LustreError: 392739:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8082.384645] LustreError: 392739:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8086.497235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8086.501684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8086.509698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8086.512720] Lustre: Skipped 1 previous similar message [ 8089.065633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.069366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8089.090446] Lustre: Skipped 1 previous similar message [ 8094.180380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.195523] Lustre: Skipped 1 previous similar message [ 8095.552815] Lustre: server umount lustre-MDT0000 complete [ 8099.297498] LustreError: 389454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.338267] LustreError: 389454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8100.543376] LustreError: 391286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374537 with bad export cookie 8458238613430662853 [ 8100.549732] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8100.554464] LustreError: 391286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8110.236689] Lustre: server umount lustre-MDT0001 complete [ 8120.717768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.110952] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8129.442203] LustreError: 393905:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8137.403339] Lustre: 394245:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.114@tcp [ 8139.322312] Lustre: server umount MGS complete [ 8150.771330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8151.203616] LustreError: 394968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8151.238916] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8151.246652] Lustre: Skipped 1 previous similar message [ 8151.314351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8157.279137] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8161.760344] LustreError: 394969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8161.795805] LustreError: 394969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8167.936933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.148363] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8168.227898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8175.324484] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8183.958995] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8189.332739] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8191.378966] Lustre: Permanently reactivating lustre-OST0000 [ 8191.404914] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8191.416256] Lustre: Skipped 2 previous similar messages [ 8218.593835] LustreError: 396707:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8229.427857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8229.675814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8241.271638] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8251.892825] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8258.534873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8258.540251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8258.542511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8258.574133] Lustre: Skipped 1 previous similar message [ 8258.593443] Lustre: Skipped 1 previous similar message [ 8259.363488] Lustre: server umount lustre-OST0000 complete [ 8268.780509] LustreError: 394963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8268.787446] LustreError: 394963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8268.835734] Lustre: server umount lustre-MDT0000 complete [ 8272.693762] LustreError: 394948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374709 with bad export cookie 8458238613430663903 [ 8272.698437] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8272.708052] LustreError: 394948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8273.196211] Lustre: server umount lustre-MDT0001 complete [ 8284.480275] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8287.897423] Key type lgssc unregistered [ 8288.227550] LNet: 399334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8288.235536] LNetError: 399334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8288.258569] LNet: Removed LNI 192.168.201.114@tcp [ 8289.183528] Key type .llcrypt unregistered [ 8289.185225] Key type ._llcrypt unregistered [ 8308.006914] Key type ._llcrypt registered [ 8308.011537] Key type .llcrypt registered [ 8308.152591] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 8323.038829] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8324.456882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8324.691745] alg: No test for adler32 (adler32-zlib) [ 8326.011890] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 8326.251880] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8327.951346] Key type lgssc registered [ 8329.343395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8367.725178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8367.745053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8369.085390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8369.125687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8369.225900] Lustre: lustre-MDT0000: new disk, initializing [ 8369.337416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8369.378505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8374.767268] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8384.943061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.003611] Lustre: 402496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8385.064306] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8385.068205] Lustre: Skipped 1 previous similar message [ 8385.135995] Lustre: lustre-MDT0001: new disk, initializing [ 8385.231772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8385.276381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8385.281698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8389.909139] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8399.443337] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8404.416798] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8410.420983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8410.610677] Lustre: lustre-OST0000: new disk, initializing [ 8410.613150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8410.617030] Lustre: 404466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8410.707290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8417.932374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8417.969405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8418.184235] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8418.224629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8428.445897] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8433.319348] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8433.580563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8439.411785] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8439.913980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8443.873321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.880117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8443.889944] Lustre: Skipped 1 previous similar message [ 8443.909897] Lustre: Skipped 1 previous similar message [ 8448.635708] Lustre: server umount lustre-OST0000 complete [ 8454.121882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.126885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8454.136719] Lustre: Skipped 1 previous similar message [ 8458.979373] Lustre: server umount lustre-MDT0000 complete [ 8462.771949] LustreError: 402489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789374900 with bad export cookie 15143333124208332802 [ 8462.780264] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8462.782458] LustreError: 402489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8463.161333] Lustre: server umount lustre-MDT0001 complete [ 8471.203255] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8472.849749] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:35:08 (1789374908) [ 8480.502349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8480.890234] LustreError: 406925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8480.903152] LustreError: 406925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8480.949859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.752810] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8486.374605] LustreError: 406926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8491.496085] LustreError: 406925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.331737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.737795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8498.986141] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8506.597912] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8513.005661] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8520.602898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8520.831761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8527.617108] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8535.421463] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8537.334254] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8541.172707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8546.792973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8546.802181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.821145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8546.825016] Lustre: Skipped 1 previous similar message [ 8549.860943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8549.861171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8549.883642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8552.431891] Lustre: server umount lustre-OST0000 complete [ 8556.512861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8556.522642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.538790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8556.540988] Lustre: Skipped 1 previous similar message [ 8562.531289] Lustre: server umount lustre-MDT0000 complete [ 8564.192302] LustreError: 407656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8564.207848] LustreError: 407656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8566.145590] LustreError: 406905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375003 with bad export cookie 15143333124208333684 [ 8566.163342] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8566.163827] LustreError: 406905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8566.614964] Lustre: server umount lustre-MDT0001 complete [ 8579.014683] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8582.044457] Key type lgssc unregistered [ 8582.496547] LNet: 410650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8582.516519] LNetError: 410650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8583.590082] LNet: Removed LNI 192.168.201.114@tcp [ 8584.555467] Key type .llcrypt unregistered [ 8584.557296] Key type ._llcrypt unregistered [ 8612.207140] Key type ._llcrypt registered [ 8612.212645] Key type .llcrypt registered [ 8612.363369] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8612.794386] Key type .llcrypt unregistered [ 8612.800893] Key type ._llcrypt unregistered [ 8630.894711] Key type ._llcrypt registered [ 8630.896798] Key type .llcrypt registered [ 8631.047954] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8632.461067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8632.538335] alg: No test for adler32 (adler32-zlib) [ 8633.926528] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 8634.162484] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8635.847316] Key type lgssc registered [ 8636.856762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8645.547652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8645.565307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8647.155717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.170923] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8657.378266] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.403298] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8661.896666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8662.204248] LustreError: 412874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.243069] LustreError: 412874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8662.253633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.621297] LustreError: 412869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.648213] LustreError: 412869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8670.829389] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8671.727915] LustreError: 412873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.761881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8676.837348] LustreError: 412874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.869415] LustreError: 412874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8679.962740] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8687.083885] LustreError: 412870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8687.102221] LustreError: 412870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8688.005961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.556885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8694.411916] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8703.972540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.980763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8703.983776] Lustre: Skipped 1 previous similar message [ 8706.103563] Lustre: server umount lustre-OST0000 complete [ 8711.144779] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8711.154989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.187477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8711.191019] Lustre: Skipped 1 previous similar message [ 8712.035284] Lustre: server umount lustre-MDT0000 complete [ 8714.208971] LustreError: 412874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.212716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8716.160981] LustreError: 412854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375153 with bad export cookie 4397583933860581996 [ 8716.171625] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8716.181177] LustreError: 412854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8716.590738] Lustre: server umount lustre-MDT0001 complete [ 8732.820295] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8736.449599] Key type lgssc unregistered [ 8736.953717] LNet: 416023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8736.963660] LNetError: 416023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8736.986968] LNet: Removed LNI 192.168.201.114@tcp [ 8738.033153] Key type .llcrypt unregistered [ 8738.036326] Key type ._llcrypt unregistered [ 8767.172269] Key type ._llcrypt registered [ 8767.174278] Key type .llcrypt registered [ 8767.300553] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8768.925579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8769.247548] alg: No test for adler32 (adler32-zlib) [ 8770.613183] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 8770.913534] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8772.727173] Key type lgssc registered [ 8774.436772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8782.383855] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8782.408882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.983528] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8803.103306] LustreError: 417411:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.114@tcp: failed processing log, type 4: rc = -110 [ 8829.791757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8837.850750] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8847.638902] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8855.768653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8856.335845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.411032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8861.677551] LustreError: 417437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.700330] LustreError: 417437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8861.827693] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8863.740953] LustreError: 418449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.766746] LustreError: 418449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8868.839220] LustreError: 418803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8868.850484] LustreError: 418803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8871.638790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.084281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8877.211639] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 8885.959927] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8892.979669] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8902.625431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8902.633084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.648414] LustreError: Skipped 1 previous similar message [ 8902.656740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8902.671103] Lustre: Skipped 1 previous similar message [ 8902.702660] Lustre: Skipped 1 previous similar message [ 8906.954680] Lustre: server umount lustre-OST0000 complete [ 8912.864341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8912.880596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8912.892124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8912.904139] Lustre: Skipped 2 previous similar messages [ 8917.941064] Lustre: server umount lustre-MDT0000 complete [ 8918.499990] LustreError: 418447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8918.526239] LustreError: 418447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8922.405463] LustreError: 417418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375359 with bad export cookie 15353478774278632398 [ 8922.426636] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8922.429975] LustreError: 417418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8923.098884] Lustre: server umount lustre-MDT0001 complete [ 8935.437917] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 8939.021860] Key type lgssc unregistered [ 8939.479831] LNet: 420982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8939.498472] LNetError: 420982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8940.518060] LNet: Removed LNI 192.168.201.114@tcp [ 8941.718187] Key type .llcrypt unregistered [ 8941.719817] Key type ._llcrypt unregistered [ 8972.478650] Key type ._llcrypt registered [ 8972.482507] Key type .llcrypt registered [ 8972.660380] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 8974.693315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8974.761065] alg: No test for adler32 (adler32-zlib) [ 8975.988381] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 8976.368644] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 8978.127212] Key type lgssc registered [ 8979.455878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8990.267797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8990.308580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.899833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8997.406533] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9002.466166] LustreError: 422399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9002.482035] LustreError: 422399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9006.641214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9007.395803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9011.851813] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9020.106915] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9027.927750] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9038.171197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9038.511438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9046.405471] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9054.918842] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9062.077735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9069.024391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9069.027645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9069.046712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9070.958410] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.14@tcp (stopping) [ 9072.613332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9072.631865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.663338] Lustre: Skipped 1 previous similar message [ 9074.636343] Lustre: server umount lustre-MDT0000 complete [ 9076.066912] LustreError: 422396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.518062] LustreError: 422379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.14@tcp arrived at 1789375514 with bad export cookie 6358903645344419681 [ 9077.729882] LustreError: 422395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.748641] LustreError: 422395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9079.668139] LustreError: 422381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789375516 with bad export cookie 6358903645344419016 [ 9079.720854] LustreError: 422381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9079.721141] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9082.848682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9082.850032] LustreError: 423417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9082.874168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9082.875402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9082.875407] Lustre: Skipped 2 previous similar messages [ 9082.974552] LustreError: 423417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9086.005821] Lustre: server umount lustre-OST0000 complete [ 9090.018895] LustreError: 423417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.041366] LustreError: 423417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9095.144527] Lustre: server umount lustre-MDT0001 complete [ 9109.132421] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 9112.871254] Key type lgssc unregistered [ 9113.198516] LNet: 426108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9113.216127] LNetError: 426108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9113.240967] LNet: Removed LNI 192.168.201.114@tcp [ 9114.398952] Key type .llcrypt unregistered [ 9114.405067] Key type ._llcrypt unregistered [ 9142.993966] Key type ._llcrypt registered [ 9142.997615] Key type .llcrypt registered [ 9143.161097] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9145.141272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9145.160027] alg: No test for adler32 (adler32-zlib) [ 9146.160654] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 9146.448313] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 9148.119164] Key type lgssc registered [ 9149.958856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9160.541373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9160.576058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.343224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9166.240526] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9172.451765] LustreError: 427551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9172.466272] LustreError: 427551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9175.361179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.725860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9180.301478] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9188.507381] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9194.285896] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9202.061468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9202.606735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.736771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9209.890693] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9218.118757] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9225.197316] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.213697] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9225.219063] Lustre: Skipped 1 previous similar message [ 9226.115191] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 9229.254349] Lustre: server umount lustre-MDT0001 complete [ 9230.307291] LustreError: 427545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9230.337956] LustreError: 427545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9235.426127] LustreError: 427550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.427766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9235.442034] LustreError: 427550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9235.455961] Lustre: Skipped 1 previous similar message [ 9235.462609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9240.545065] LustreError: 427551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.563823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9241.235264] Lustre: server umount lustre-OST0000 complete [ 9246.145704] Lustre: server umount lustre-MDT0000 complete [ 9261.837378] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 9265.055556] Key type lgssc unregistered [ 9265.440718] LNet: 431234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9265.455156] LNetError: 431234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9265.476139] LNet: Removed LNI 192.168.201.114@tcp [ 9266.587877] Key type .llcrypt unregistered [ 9266.590988] Key type ._llcrypt unregistered [ 9292.012424] Key type ._llcrypt registered [ 9292.015348] Key type .llcrypt registered [ 9292.166247] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [ 9308.209937] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9309.361076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9309.519338] alg: No test for adler32 (adler32-zlib) [ 9310.701705] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 9310.977113] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 9312.687270] Key type lgssc registered [ 9313.974639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9349.495978] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9364.035430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9364.071442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.359413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9365.399889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9365.473971] Lustre: lustre-MDT0000: new disk, initializing [ 9365.518232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.528884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9369.727568] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9383.919247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9384.230639] Lustre: lustre-OST0000: new disk, initializing [ 9384.236611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9384.242740] Lustre: Skipped 1 previous similar message [ 9384.261569] Lustre: 436268:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9384.352823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9389.411518] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9391.213271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9391.229662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9391.252807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9397.691869] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9410.939598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.020713] Lustre: 435251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9411.094579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9411.174514] Lustre: lustre-MDT0001: new disk, initializing [ 9411.300551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9411.351474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9411.364423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9415.027542] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9424.472626] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9424.781861] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9429.329729] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9429.597794] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9434.483684] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.14@tcp (stopping) [ 9439.213847] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.220831] Lustre: Skipped 1 previous similar message [ 9439.221110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9439.240601] Lustre: Skipped 1 previous similar message [ 9440.910631] Lustre: server umount lustre-MDT0001 complete [ 9449.457801] LustreError: 435259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9449.499416] LustreError: 435259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9454.567513] LustreError: 435262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9454.578805] LustreError: 435262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9459.681898] LustreError: 437595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.684317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9459.723124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9459.725771] Lustre: Skipped 1 previous similar message [ 9464.812670] LustreError: 435262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9464.925254] Lustre: server umount lustre-OST0000 complete [ 9484.255271] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9484.559221] Lustre: server umount lustre-MDT0000 complete [ 9500.075612] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 9504.292602] Key type lgssc unregistered [ 9504.785866] LNet: 440017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9504.801812] LNetError: 440017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9504.826724] LNet: Removed LNI 192.168.201.114@tcp [ 9506.308854] Key type .llcrypt unregistered [ 9506.313062] Key type ._llcrypt unregistered [ 9550.290036] Key type ._llcrypt registered [ 9550.292202] Key type .llcrypt registered [ 9550.351103] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [ 9550.529628] Key type .llcrypt unregistered [ 9550.532656] Key type ._llcrypt unregistered [ 9568.436663] Key type ._llcrypt registered [ 9568.439263] Key type .llcrypt registered [ 9568.613277] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [ 9570.013557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9570.080760] alg: No test for adler32 (adler32-zlib) [ 9571.473574] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [ 9571.756258] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [ 9573.543241] Key type lgssc registered [ 9575.508141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9606.704191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9606.748928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.022971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9608.069274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9608.261705] Lustre: lustre-MDT0000: new disk, initializing [ 9608.401389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.442456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9613.558966] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9625.989842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9626.148731] Lustre: 443403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9626.273019] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9626.276800] Lustre: Skipped 1 previous similar message [ 9626.397640] Lustre: lustre-MDT0001: new disk, initializing [ 9626.544275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9626.613471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9626.626449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9631.440451] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9640.177284] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9645.934173] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9654.251337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9654.637398] Lustre: lustre-OST0000: new disk, initializing [ 9654.640779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9654.647399] Lustre: 445371:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9654.751892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9661.344560] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9664.611914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9664.616605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9664.869965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9672.425142] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9680.354674] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9680.357304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9680.371360] Lustre: Skipped 2 previous similar messages [ 9680.384468] Lustre: Skipped 2 previous similar messages [ 9685.476604] LustreError: 443414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9685.492770] LustreError: 443414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9685.732712] Lustre: server umount lustre-MDT0000 complete [ 9690.638877] LustreError: 443414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9690.646088] LustreError: 443414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9690.710362] LustreError: 443394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376128 with bad export cookie 10750084742116189045 [ 9690.721723] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9691.164712] Lustre: server umount lustre-MDT0001 complete [ 9706.304358] Lustre: server umount lustre-OST0000 complete [ 9718.176981] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:55:53 (1789376153) [ 9726.855048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.436750] LustreError: 447399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9727.539355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9732.585735] LustreError: 447400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9732.646908] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9737.711164] LustreError: 447399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9741.559654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9742.047524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9747.493709] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9756.312714] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9762.485034] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9782.581509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9782.852417] Lustre: Found index 0 for lustre-OST0000, updating log [ 9783.112552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9789.369241] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [ 9797.713373] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9799.540123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9809.890878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.911056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9810.982161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9810.993919] Lustre: Skipped 2 previous similar messages [ 9816.449582] Lustre: server umount lustre-OST0000 complete [ 9834.464820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9834.489981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9834.515674] Lustre: Skipped 1 previous similar message [ 9834.527749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9837.542284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9838.561605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9838.572283] Lustre: Skipped 2 previous similar messages [ 9839.592473] LustreError: 447396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9839.609868] LustreError: 447396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9839.795373] Lustre: server umount lustre-MDT0000 complete [ 9844.064968] LustreError: 447381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789376281 with bad export cookie 10750084742116190431 [ 9844.074852] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9844.099484] LustreError: 447381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9844.387275] Lustre: server umount lustre-MDT0001 complete [ 9847.266192] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp [ 9855.730679] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:58:10 (1789376290) [10495.014263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.917063] LustreError: 453828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.069724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10501.095310] LustreError: 453829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10501.127597] LustreError: 453829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10501.583697] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10506.213297] LustreError: 453828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10511.901502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.355936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10518.420226] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10528.221865] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10534.876755] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10551.839266] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp [10555.702738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10556.534152] Lustre: Found index 0 for lustre-OST0000, updating log [10556.865900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10565.022315] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10572.813695] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10577.574305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10577.609297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10577.611916] Lustre: Skipped 1 previous similar message [10583.119969] Lustre: server umount lustre-OST0000 complete [10589.474196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10589.479182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10589.487474] Lustre: Skipped 1 previous similar message [10589.522945] Lustre: Skipped 1 previous similar message [10591.522977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10591.535477] Lustre: Skipped 2 previous similar messages [10594.804596] Lustre: server umount lustre-MDT0000 complete [10595.628402] LustreError: 453828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10595.655108] LustreError: 453828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10599.468067] LustreError: 453807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377036 with bad export cookie 10750084742116191523 [10599.489757] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10599.973974] Lustre: server umount lustre-MDT0001 complete [10612.320486] LNet: 1 local NIs in recovery (showing 1): 192.168.201.114@tcp [10634.345896] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [10647.349261] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [10692.480601] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:12:06 (1789377126) [10720.325177] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [10724.841687] Key type lgssc unregistered [10725.113803] LNet: 460440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10725.117857] LNetError: 460440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10819.383647] LNet: Removed LNI 192.168.201.114@tcp [10820.531620] Key type .llcrypt unregistered [10820.537988] Key type ._llcrypt unregistered [10839.753731] Key type ._llcrypt registered [10839.755815] Key type .llcrypt registered [10839.876592] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [10841.180611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10841.408640] alg: No test for adler32 (adler32-zlib) [10842.735662] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [10843.112736] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [10844.888350] Key type lgssc registered [10846.117145] Lustre: Echo OBD driver; http://www.lustre.org/ [10850.967728] LNet: Added LNI 192.168.201.114@tcp100 [8/256/0/180] [10851.884237] LNet: Added LNI 192.168.201.114@tcp101 [8/256/0/180] [10852.705173] LNet: Added LNI 192.168.201.114@tcp102 [8/256/0/180] [10853.351211] LNet: Added LNI 192.168.201.114@tcp103 [8/256/0/180] [10853.989823] LNet: Added LNI 192.168.201.114@tcp104 [8/256/0/180] [10854.752518] LNet: Added LNI 192.168.201.114@tcp105 [8/256/0/180] [10855.799342] LNet: Added LNI 192.168.201.114@tcp106 [8/256/0/180] [10856.555550] LNet: Added LNI 192.168.201.114@tcp107 [8/256/0/180] [10857.226666] LNet: Added LNI 192.168.201.114@tcp108 [8/256/0/180] [10857.807681] LNet: Added LNI 192.168.201.114@tcp109 [8/256/0/180] [10858.655875] LNet: Added LNI 192.168.201.114@tcp110 [8/256/0/180] [10859.678206] LNet: Added LNI 192.168.201.114@tcp111 [8/256/0/180] [10860.586366] LNet: Added LNI 192.168.201.114@tcp112 [8/256/0/180] [10861.400847] LNet: Added LNI 192.168.201.114@tcp113 [8/256/0/180] [10862.227303] LNet: Added LNI 192.168.201.114@tcp114 [8/256/0/180] [10863.145857] LNet: Added LNI 192.168.201.114@tcp115 [8/256/0/180] [10864.035480] LNet: Added LNI 192.168.201.114@tcp116 [8/256/0/180] [10864.750373] LNet: Added LNI 192.168.201.114@tcp117 [8/256/0/180] [10865.556245] LNet: Added LNI 192.168.201.114@tcp118 [8/256/0/180] [10866.418592] LNet: Added LNI 192.168.201.114@tcp119 [8/256/0/180] [10867.177997] LNet: Added LNI 192.168.201.114@tcp120 [8/256/0/180] [10867.955420] LNet: Added LNI 192.168.201.114@tcp121 [8/256/0/180] [10868.815372] LNet: Added LNI 192.168.201.114@tcp122 [8/256/0/180] [10869.896381] LNet: Added LNI 192.168.201.114@tcp123 [8/256/0/180] [10870.941178] LNet: Added LNI 192.168.201.114@tcp124 [8/256/0/180] [10872.039094] LNet: Added LNI 192.168.201.114@tcp125 [8/256/0/180] [10873.151770] LNet: Added LNI 192.168.201.114@tcp126 [8/256/0/180] [10874.468535] LNet: Added LNI 192.168.201.114@tcp127 [8/256/0/180] [10875.955789] LNet: Added LNI 192.168.201.114@tcp128 [8/256/0/180] [10877.222694] LNet: Added LNI 192.168.201.114@tcp129 [8/256/0/180] [10877.986056] LNet: Added LNI 192.168.201.114@tcp130 [8/256/0/180] [10878.812793] LNet: Added LNI 192.168.201.114@tcp131 [8/256/0/180] [10879.766147] LNet: Added LNI 192.168.201.114@tcp132 [8/256/0/180] [10880.551996] LNet: Added LNI 192.168.201.114@tcp133 [8/256/0/180] [10881.546829] LNet: Added LNI 192.168.201.114@tcp134 [8/256/0/180] [10882.656324] LNet: Added LNI 192.168.201.114@tcp135 [8/256/0/180] [10893.115093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10893.149115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.275638] Lustre: 463466:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [10894.378325] LNetError: 460922:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [10894.393261] LNetError: Protocol error connecting to 192.168.201.114@tcp114 at host 192.168.201.114:988: is it running a compatible version of Lustre? [10894.429278] LNetError: Skipped 8 previous similar messages [10894.520274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10894.559429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10894.766955] Lustre: lustre-MDT0000: new disk, initializing [10894.947729] Lustre: 463521:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [10894.964179] Lustre: 463521:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [10895.024692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10895.059392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10895.327381] LNet: 6 local NIs in recovery (showing 6): 192.168.201.114@tcp121, 192.168.201.114@tcp124, 192.168.201.114@tcp125, 192.168.201.114@tcp127, 192.168.201.114@tcp128, 192.168.201.114@tcp129 [10895.329134] LNetError: 463487:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [10895.374294] LNetError: 463487:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [10895.394796] LNetError: Protocol error connecting to 192.168.201.114@tcp101 at host 192.168.201.114:988: is it running a compatible version of Lustre? [10895.410917] LNetError: Skipped 4 previous similar messages [10900.250753] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10914.970889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10915.404681] Lustre: 463481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10915.609813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10915.624091] Lustre: Skipped 1 previous similar message [10915.800761] Lustre: lustre-MDT0001: new disk, initializing [10915.875256] Lustre: 464285:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [10915.894392] Lustre: 464285:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [10916.144460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10916.193053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10916.209261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10922.551876] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10922.630476] Lustre: 464545:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [10922.653648] Lustre: 464545:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10925.026188] LNetError: 463492:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [10925.034698] LNetError: 463492:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [10925.040176] LNetError: Protocol error connecting to 192.168.201.114@tcp114 at host 192.168.201.114:988: is it running a compatible version of Lustre? [10928.102135] LNetError: 463486:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [10928.132305] LNetError: 463486:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [10928.157272] LNetError: Protocol error connecting to 192.168.201.114@tcp125 at host 192.168.201.114:988: is it running a compatible version of Lustre? [10928.184838] LNetError: Skipped 1 previous similar message [10931.922445] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10932.193745] LNetError: 460924:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [10932.206102] LNetError: 460924:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [10932.214768] LNetError: Protocol error connecting to 192.168.201.114@tcp118 at host 192.168.201.114:988: is it running a compatible version of Lustre? [10932.236662] LNetError: Skipped 2 previous similar messages [10937.390510] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10941.486052] LNetError: 460923:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [10941.503324] LNetError: 460923:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [10941.517386] LNetError: Protocol error connecting to 192.168.201.114@tcp125 at host 192.168.201.114:988: is it running a compatible version of Lustre? [10941.530797] LNetError: Skipped 5 previous similar messages [10944.683137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.072782] Lustre: lustre-OST0000: new disk, initializing [10945.086637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10945.108232] Lustre: 465455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10945.371890] Lustre: 465460:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.114@tcp: can't add NID 192.168.201.114@tcp130: rc = -75 [10945.397707] Lustre: 465460:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [10945.532348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10952.420285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10952.437757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10952.736733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10955.920615] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [10957.856277] LNetError: 460921:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [10957.887051] LNetError: 460921:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [10957.896394] LNetError: Protocol error connecting to 192.168.201.114@tcp118 at host 192.168.201.114:988: is it running a compatible version of Lustre? [10957.912274] LNetError: Skipped 11 previous similar messages [10968.836594] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10978.282630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10978.289297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10978.290937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10978.307804] Lustre: Skipped 1 previous similar message [10978.333177] LustreError: Skipped 1 previous similar message [10982.701141] Lustre: server umount lustre-OST0000 complete [10988.000832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10988.025394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.052741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.055425] Lustre: Skipped 1 previous similar message [10992.675725] LNetError: 463486:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.114 [10992.700776] LNetError: 463486:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [10992.715170] LNetError: Protocol error connecting to 192.168.201.114@tcp125 at host 192.168.201.114:988: is it running a compatible version of Lustre? [10992.726165] LNetError: Skipped 13 previous similar messages [10993.729207] Lustre: server umount lustre-MDT0000 complete [10998.223994] LustreError: 463476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789377435 with bad export cookie 12291111635135332900 [10998.233124] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10998.243150] LustreError: 463476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10998.721988] Lustre: server umount lustre-MDT0001 complete [11012.372837] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11015.264595] Key type lgssc unregistered [11015.680962] LNet: 467370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11015.689589] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11015.695128] LNet: Removed LNI 192.168.201.114@tcp [11016.740391] LNet: Removed LNI 192.168.201.114@tcp100 [11016.768389] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11016.776803] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11016.799770] LNet: Removed LNI 192.168.201.114@tcp101 [11016.840584] LNet: Removed LNI 192.168.201.114@tcp102 [11017.887346] LNet: Removed LNI 192.168.201.114@tcp103 [11017.907352] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11017.925738] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11017.948979] LNet: Removed LNI 192.168.201.114@tcp104 [11018.975142] LNet: Removed LNI 192.168.201.114@tcp105 [11019.999647] LNet: Removed LNI 192.168.201.114@tcp106 [11020.001501] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11020.040922] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11020.049347] LNet: Removed LNI 192.168.201.114@tcp107 [11021.087141] LNet: Removed LNI 192.168.201.114@tcp108 [11022.111188] LNet: Removed LNI 192.168.201.114@tcp109 [11023.135434] LNet: Removed LNI 192.168.201.114@tcp110 [11024.164336] LNet: Removed LNI 192.168.201.114@tcp111 [11024.173705] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11024.189392] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11024.201855] LNet: Removed LNI 192.168.201.114@tcp112 [11025.267434] LNet: Removed LNI 192.168.201.114@tcp113 [11026.335259] LNet: Removed LNI 192.168.201.114@tcp114 [11027.361851] LNet: Removed LNI 192.168.201.114@tcp115 [11028.383174] LNet: Removed LNI 192.168.201.114@tcp116 [11029.407155] LNet: Removed LNI 192.168.201.114@tcp117 [11030.431163] LNet: Removed LNI 192.168.201.114@tcp118 [11031.460289] LNet: Removed LNI 192.168.201.114@tcp119 [11032.480760] LNet: Removed LNI 192.168.201.114@tcp120 [11032.494913] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11032.517080] LNetError: 467370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11032.525071] LNet: Removed LNI 192.168.201.114@tcp121 [11033.567428] LNet: Removed LNI 192.168.201.114@tcp122 [11034.591756] LNet: Removed LNI 192.168.201.114@tcp123 [11035.615141] LNet: Removed LNI 192.168.201.114@tcp124 [11036.639155] LNet: Removed LNI 192.168.201.114@tcp125 [11037.663170] LNet: Removed LNI 192.168.201.114@tcp126 [11038.687161] LNet: Removed LNI 192.168.201.114@tcp127 [11039.711145] LNet: Removed LNI 192.168.201.114@tcp128 [11040.735294] LNet: Removed LNI 192.168.201.114@tcp129 [11040.737777] LNet: Removed LNI 192.168.201.114@tcp130 [11040.741153] LNet: Removed LNI 192.168.201.114@tcp131 [11040.747488] LNet: Removed LNI 192.168.201.114@tcp132 [11040.751300] LNet: Removed LNI 192.168.201.114@tcp133 [11040.753604] LNet: Removed LNI 192.168.201.114@tcp134 [11040.766578] LNet: Removed LNI 192.168.201.114@tcp135 [11041.544418] Key type .llcrypt unregistered [11041.547815] Key type ._llcrypt unregistered [11068.280719] Key type ._llcrypt registered [11068.282868] Key type .llcrypt registered [11068.406959] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [11086.389665] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11088.110218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11088.148382] alg: No test for adler32 (adler32-zlib) [11089.144727] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [11089.358756] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11091.087952] Key type lgssc registered [11092.216694] Lustre: Echo OBD driver; http://www.lustre.org/ [11136.707816] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:19:31 (1789377571) [11153.898749] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11156.722131] Key type lgssc unregistered [11156.976608] LNet: 471454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11156.980995] LNetError: 471454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11156.992715] LNet: Removed LNI 192.168.201.114@tcp [11157.631618] Key type .llcrypt unregistered [11157.633637] Key type ._llcrypt unregistered [11176.129978] Key type ._llcrypt registered [11176.134063] Key type .llcrypt registered [11176.316363] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11177.909044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11177.936311] alg: No test for adler32 (adler32-zlib) [11179.173561] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [11179.674424] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11181.391310] Key type lgssc registered [11183.229443] Lustre: Echo OBD driver; http://www.lustre.org/ [11194.638436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11194.661234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11195.917754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11195.935776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11195.988167] Lustre: lustre-MDT0000: new disk, initializing [11196.040852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11196.053798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11200.686837] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11213.083879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.283778] Lustre: 472663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11213.374163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11213.384326] Lustre: Skipped 1 previous similar message [11213.478213] Lustre: lustre-MDT0001: new disk, initializing [11213.557258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11213.595309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11213.604774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11218.713585] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11229.456689] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11234.751461] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11240.567267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11240.869276] Lustre: lustre-OST0000: new disk, initializing [11240.872611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11240.879289] Lustre: 474633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11240.981323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11246.646662] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11249.233212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11249.243993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11249.389044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11256.517877] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11260.188833] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing lnet_if_list [11266.307506] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11266.378571] LNet: Added LNI 192.168.201.114@tcp10 [8/256/0/180] [11277.773123] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11277.809611] LNet: Added LNI 192.168.201.114@tcp11 [8/256/0/180] [11283.976045] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11284.005765] LNet: Added LNI 192.168.201.114@tcp12 [8/256/0/180] [11288.435671] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11288.462677] LNet: Added LNI 192.168.201.114@tcp13 [8/256/0/180] [11295.364792] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11295.393458] LNet: Added LNI 192.168.201.114@tcp14 [8/256/0/180] [11301.709317] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11301.749044] LNet: Added LNI 192.168.201.114@tcp15 [8/256/0/180] [11309.919913] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11310.014198] LNet: Added LNI 192.168.201.114@tcp16 [8/256/0/180] [11319.230919] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11319.286703] LNet: Added LNI 192.168.201.114@tcp17 [8/256/0/180] [11325.850363] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11325.865445] LNet: Added LNI 192.168.201.114@tcp18 [8/256/0/180] [11331.127063] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11331.154438] LNet: Added LNI 192.168.201.114@tcp19 [8/256/0/180] [11337.110748] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11337.149775] LNet: Added LNI 192.168.201.114@tcp20 [8/256/0/180] [11344.336436] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11344.374879] LNet: Added LNI 192.168.201.114@tcp21 [8/256/0/180] [11350.551456] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11350.583607] LNet: Added LNI 192.168.201.114@tcp22 [8/256/0/180] [11355.716569] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11355.741421] LNet: Added LNI 192.168.201.114@tcp23 [8/256/0/180] [11361.397827] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11361.466676] LNet: Added LNI 192.168.201.114@tcp24 [8/256/0/180] [11369.274288] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11369.310392] LNet: Added LNI 192.168.201.114@tcp25 [8/256/0/180] [11380.876961] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11380.914622] LNet: Added LNI 192.168.201.114@tcp26 [8/256/0/180] [11389.701681] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11389.728832] LNet: Added LNI 192.168.201.114@tcp27 [8/256/0/180] [11396.449618] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11396.463554] LNet: Added LNI 192.168.201.114@tcp28 [8/256/0/180] [11402.224211] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11402.245804] LNet: Added LNI 192.168.201.114@tcp29 [8/256/0/180] [11409.911419] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11409.962627] LNet: Added LNI 192.168.201.114@tcp30 [8/256/0/180] [11419.011433] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11419.037465] LNet: Added LNI 192.168.201.114@tcp31 [8/256/0/180] [11427.959532] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11428.012452] LNet: Added LNI 192.168.201.114@tcp32 [8/256/0/180] [11435.740858] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11435.759651] LNet: Added LNI 192.168.201.114@tcp33 [8/256/0/180] [11442.499541] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11442.535767] LNet: Added LNI 192.168.201.114@tcp34 [8/256/0/180] [11450.052704] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11450.083779] LNet: Added LNI 192.168.201.114@tcp35 [8/256/0/180] [11458.216504] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11458.258372] LNet: Added LNI 192.168.201.114@tcp36 [8/256/0/180] [11464.131470] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11464.156532] LNet: Added LNI 192.168.201.114@tcp37 [8/256/0/180] [11470.170216] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11470.193781] LNet: Added LNI 192.168.201.114@tcp38 [8/256/0/180] [11478.694986] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11478.741428] LNet: Added LNI 192.168.201.114@tcp39 [8/256/0/180] [11489.249780] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11489.304465] LNet: Added LNI 192.168.201.114@tcp40 [8/256/0/180] [11497.380650] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11497.409879] LNet: Added LNI 192.168.201.114@tcp41 [8/256/0/180] [11505.181847] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11505.203650] LNet: Added LNI 192.168.201.114@tcp42 [8/256/0/180] [11511.184682] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11511.203919] LNet: Added LNI 192.168.201.114@tcp43 [8/256/0/180] [11522.473456] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11522.527916] LNet: Added LNI 192.168.201.114@tcp44 [8/256/0/180] [11532.384477] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11532.431629] LNet: Added LNI 192.168.201.114@tcp45 [8/256/0/180] [11539.754037] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11539.766715] LNet: Added LNI 192.168.201.114@tcp46 [8/256/0/180] [11548.908190] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11548.931278] LNet: Added LNI 192.168.201.114@tcp47 [8/256/0/180] [11560.645683] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11560.709927] LNet: Added LNI 192.168.201.114@tcp48 [8/256/0/180] [11569.755684] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11569.866721] LNet: Added LNI 192.168.201.114@tcp49 [8/256/0/180] [11578.516724] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11578.536117] LNet: Added LNI 192.168.201.114@tcp50 [8/256/0/180] [11597.795046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11597.798529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11597.806973] Lustre: Skipped 1 previous similar message [11597.812176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11597.819827] Lustre: Skipped 2 previous similar messages [11602.158235] Lustre: server umount lustre-MDT0000 complete [11605.703328] LustreError: 473459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378043 with bad export cookie 530844706830271186 [11605.707610] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11605.716368] LustreError: 473459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11606.064343] Lustre: server umount lustre-MDT0001 complete [11624.416847] Lustre: 471938:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378045/real 1789378045] req@0000000043df6d4c x1876298423257600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11624.433645] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11624.451700] Lustre: Skipped 1 previous similar message [11625.073398] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11626.463960] Lustre: 471936:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378047/real 1789378047] req@00000000f15e513f x1876298423257728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11629.536326] Lustre: 471935:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378050/real 1789378050] req@0000000002202ff6 x1876298423257856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378066 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11632.610315] Lustre: 471937:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378053/real 1789378053] req@000000003bb18af2 x1876298423258112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11634.253834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11636.703153] Lustre: 471938:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789378058/real 1789378058] req@000000005d030ba2 x1876298423258240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789378074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11652.535972] LustreError: 474627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11652.559712] LustreError: 474627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11652.588264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11653.616081] LustreError: 474628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11653.632962] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11654.634540] LustreError: 474627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11657.292614] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11659.749220] LustreError: 474628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11664.868766] LustreError: 474628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11666.353568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11666.741523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11668.780961] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11670.787385] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11678.661975] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11684.768977] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11709.925544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11709.950712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11709.956528] Lustre: Skipped 1 previous similar message [11715.057090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11715.062128] Lustre: Skipped 1 previous similar message [11715.761771] Lustre: server umount lustre-OST0000 complete [11720.170075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11720.178227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11720.183283] Lustre: Skipped 2 previous similar messages [11720.204286] Lustre: Skipped 1 previous similar message [11725.289513] LustreError: 498885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.310058] LustreError: 498885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11725.386184] Lustre: server umount lustre-MDT0000 complete [11728.497097] LustreError: 472656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378165 with bad export cookie 530844706830273825 [11728.509236] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11728.509609] LustreError: 472656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11728.761608] Lustre: server umount lustre-MDT0001 complete [11739.860883] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11743.533508] Key type lgssc unregistered [11743.782961] LNet: 501681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11743.792512] LNetError: 501681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11744.803424] LNet: Removed LNI 192.168.201.114@tcp [11744.805293] LNetError: 501681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11744.811062] LNet: Removed LNI 192.168.201.114@tcp10 [11744.815380] LNet: Removed LNI 192.168.201.114@tcp11 [11744.825856] LNet: Removed LNI 192.168.201.114@tcp12 [11744.835420] LNet: Removed LNI 192.168.201.114@tcp13 [11744.838704] LNet: Removed LNI 192.168.201.114@tcp14 [11744.841301] LNet: Removed LNI 192.168.201.114@tcp15 [11744.846761] LNet: Removed LNI 192.168.201.114@tcp16 [11744.849216] LNet: Removed LNI 192.168.201.114@tcp17 [11744.854574] LNet: Removed LNI 192.168.201.114@tcp18 [11744.862743] LNet: Removed LNI 192.168.201.114@tcp19 [11744.866926] LNet: Removed LNI 192.168.201.114@tcp20 [11744.870691] LNet: Removed LNI 192.168.201.114@tcp21 [11744.872530] LNet: Removed LNI 192.168.201.114@tcp22 [11744.873976] LNet: Removed LNI 192.168.201.114@tcp23 [11744.876563] LNet: Removed LNI 192.168.201.114@tcp24 [11744.880867] LNet: Removed LNI 192.168.201.114@tcp25 [11744.891854] LNet: Removed LNI 192.168.201.114@tcp26 [11744.900477] LNet: Removed LNI 192.168.201.114@tcp27 [11744.903453] LNet: Removed LNI 192.168.201.114@tcp28 [11744.906174] LNet: Removed LNI 192.168.201.114@tcp29 [11744.909224] LNet: Removed LNI 192.168.201.114@tcp30 [11744.912447] LNet: Removed LNI 192.168.201.114@tcp31 [11744.914877] LNet: Removed LNI 192.168.201.114@tcp32 [11744.918835] LNet: Removed LNI 192.168.201.114@tcp33 [11744.920498] LNet: Removed LNI 192.168.201.114@tcp34 [11744.923737] LNet: Removed LNI 192.168.201.114@tcp35 [11744.935050] LNet: Removed LNI 192.168.201.114@tcp36 [11744.937934] LNet: Removed LNI 192.168.201.114@tcp37 [11744.944529] LNet: Removed LNI 192.168.201.114@tcp38 [11744.950853] LNet: Removed LNI 192.168.201.114@tcp39 [11744.955932] LNet: Removed LNI 192.168.201.114@tcp40 [11744.958418] LNet: Removed LNI 192.168.201.114@tcp41 [11744.961167] LNet: Removed LNI 192.168.201.114@tcp42 [11744.962721] LNet: Removed LNI 192.168.201.114@tcp43 [11744.964154] LNet: Removed LNI 192.168.201.114@tcp44 [11744.971215] LNet: Removed LNI 192.168.201.114@tcp45 [11744.974192] LNet: Removed LNI 192.168.201.114@tcp46 [11744.977359] LNet: Removed LNI 192.168.201.114@tcp47 [11744.984802] LNet: Removed LNI 192.168.201.114@tcp48 [11744.989996] LNet: Removed LNI 192.168.201.114@tcp49 [11745.007037] LNet: Removed LNI 192.168.201.114@tcp50 [11745.831205] Key type .llcrypt unregistered [11745.833184] Key type ._llcrypt unregistered [11771.005569] Key type ._llcrypt registered [11771.007841] Key type .llcrypt registered [11771.181611] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [11788.236371] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11789.735584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11789.942632] alg: No test for adler32 (adler32-zlib) [11791.474401] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [11791.926457] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11793.705737] Key type lgssc registered [11795.517166] Lustre: Echo OBD driver; http://www.lustre.org/ [11839.258658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11839.299052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11840.668649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11840.712787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11840.823557] Lustre: lustre-MDT0000: new disk, initializing [11840.930900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11840.958643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11846.081636] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11859.061518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11859.245296] Lustre: 504879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11859.317908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11859.328656] Lustre: Skipped 1 previous similar message [11859.523593] Lustre: lustre-MDT0001: new disk, initializing [11859.667734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11859.698598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11859.712837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11865.045571] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11875.751916] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11881.467883] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11888.803322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11889.064740] Lustre: *** cfs_fail_loc=90f, val=0*** [11889.128525] Lustre: lustre-OST0000: new disk, initializing [11889.131361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11889.136945] Lustre: 506893:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11889.240364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11895.894058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11895.904988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11895.960520] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [11896.092797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11908.077970] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11921.909183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11921.909737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11921.915161] LustreError: Skipped 1 previous similar message [11921.917506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11921.983499] Lustre: Skipped 1 previous similar message [11924.681783] Lustre: server umount lustre-OST0000 complete [11931.617476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11931.622432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11931.642426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11931.677097] Lustre: Skipped 1 previous similar message [11935.717601] Lustre: server umount lustre-MDT0000 complete [11938.881045] LustreError: 504870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378376 with bad export cookie 4357240519755178373 [11938.890184] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11938.897762] LustreError: 504870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11939.509716] Lustre: server umount lustre-MDT0001 complete [11949.484809] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [11952.301820] Key type lgssc unregistered [11952.679871] LNet: 508797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11952.693391] LNetError: 508797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11952.748301] LNet: Removed LNI 192.168.201.114@tcp [11953.615672] Key type .llcrypt unregistered [11953.618520] Key type ._llcrypt unregistered [11974.799765] Key type ._llcrypt registered [11974.805546] Key type .llcrypt registered [11974.989097] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [11977.102844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11977.131664] alg: No test for adler32 (adler32-zlib) [11978.610359] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [11978.813202] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [11980.479259] Key type lgssc registered [11983.426896] Lustre: Echo OBD driver; http://www.lustre.org/ [11993.757327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11993.814290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11995.541341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12001.151677] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12004.839109] LustreError: 510068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.870467] LustreError: 510068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12009.962397] LustreError: 510067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12011.492325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12011.967048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12017.061841] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12023.971975] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12030.972107] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12037.671982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12037.879899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12046.596024] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12055.911730] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12069.823103] loop0: detected capacity change from 0 to 204800000 [12070.036660] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12070.056343] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12070.070111] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12076.273860] loop0: detected capacity change from 0 to 204800000 [12076.595955] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12076.613289] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12076.627025] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12083.155395] loop0: detected capacity change from 0 to 204800000 [12089.444223] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12089.718814] Lustre: fs15246-OST0000: new disk, initializing [12089.723983] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12089.737355] Lustre: 513663:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12089.837554] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12099.998471] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12114.066312] loop1: detected capacity change from 0 to 204800000 [12121.452033] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12121.634333] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12121.716528] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12121.782188] Lustre: fs15246-MDT0000: new disk, initializing [12121.957063] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12121.972032] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12122.052155] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12123.106653] Lustre: 514788:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000054c99eab x1876299261988736/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789378576 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12129.669427] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12132.221364] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12132.239450] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12132.295731] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12153.318601] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12153.339711] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12158.450403] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12158.938408] Lustre: server umount fs15246-MDT0000 complete [12163.557401] LustreError: 510064:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.685338] LustreError: 511219:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12170.893353] Lustre: server umount fs15246-OST0000 complete [12188.652682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12188.674675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12188.698081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12193.715223] Lustre: server umount lustre-OST0000 complete [12199.393387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12199.417045] Lustre: Skipped 1 previous similar message [12199.429468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12199.439730] Lustre: Skipped 3 previous similar messages [12204.509589] Lustre: server umount lustre-MDT0000 complete [12204.514401] LustreError: 510067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12204.546856] LustreError: 510067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12208.200500] LustreError: 514782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378645 with bad export cookie 7239277867340100730 [12208.201519] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12208.209480] LustreError: 514782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12208.496863] Lustre: server umount lustre-MDT0001 complete [12220.392917] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12223.915447] Key type lgssc unregistered [12224.284410] LNet: 517884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12224.295724] LNetError: 517884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12225.330479] LNet: Removed LNI 192.168.201.114@tcp [12226.217489] Key type .llcrypt unregistered [12226.220595] Key type ._llcrypt unregistered [12246.835467] Key type ._llcrypt registered [12246.836675] Key type .llcrypt registered [12246.903649] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12248.869040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12249.268050] alg: No test for adler32 (adler32-zlib) [12250.651197] Lustre: Lustre: Build Version: 2.17.58_6_ge9bcf18 [12251.027094] LNet: Added LNI 192.168.201.114@tcp [8/256/0/180] [12252.727175] Key type lgssc registered [12254.340450] Lustre: Echo OBD driver; http://www.lustre.org/ [12305.443222] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [12319.315832] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12364.968406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12365.021092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12366.694495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12366.760358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12366.953957] Lustre: lustre-MDT0000: new disk, initializing [12367.154149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12367.180211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12372.739773] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12386.050330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12386.260924] Lustre: 522353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12386.390641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12386.394641] Lustre: Skipped 1 previous similar message [12386.560176] Lustre: lustre-MDT0001: new disk, initializing [12386.708208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12386.751650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12386.773874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12391.633676] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12403.444991] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12412.499965] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12419.323353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12419.681810] Lustre: lustre-OST0000: new disk, initializing [12419.689316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12419.696665] Lustre: 524322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12419.825560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12425.307122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12425.326246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12425.538082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12427.796939] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_default_debug -1 all [12439.604458] Lustre: DEBUG MARKER: oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12445.099493] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12445.411346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12451.243428] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12451.594552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12456.421502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12456.425709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12456.464927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12456.472367] Lustre: Skipped 1 previous similar message [12459.827532] Lustre: server umount lustre-OST0000 complete [12466.666462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12466.692292] Lustre: Skipped 1 previous similar message [12466.703211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12466.711503] Lustre: Skipped 1 previous similar message [12470.953925] Lustre: server umount lustre-MDT0000 complete [12477.648430] LustreError: 522343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789378914 with bad export cookie 9647645533092031602 [12477.675131] LustreError: MGC192.168.201.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12478.690905] Lustre: server umount lustre-MDT0001 complete [12506.257935] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing set_hostid [12519.574666] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing load_modules_local [12557.328813] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12300 sec ========= 05:43:12 (1789378992) [12559.038483] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:43:14 (1789378994) === [12563.543530] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:43:18 (1789378998) === [12609.792531] Lustre: DEBUG MARKER: oleg114-server.virtnet: executing unload_modules_local [12613.281328] Key type lgssc unregistered [12613.706048] LNet: 530969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12613.712180] LNetError: 530969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12613.720777] LNet: Removed LNI 192.168.201.114@tcp [12614.576980] Key type .llcrypt unregistered [12614.578421] Key type ._llcrypt unregistered