[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481014540 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002273] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007565] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.023185] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028885] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030205] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032128] *** VALIDATE tmpfs *** [ 0.034337] *** VALIDATE proc *** [ 0.036219] *** VALIDATE cgroup *** [ 0.037013] *** VALIDATE cgroup2 *** [ 0.038288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040079] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042023] Spectre V2 : User space: Vulnerable [ 0.043078] Speculative Store Bypass: Vulnerable [ 0.046458] debug: unmapping init [mem 0xffffffffa0259000-0xffffffffa0260fff] [ 0.048242] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049663] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050026] ... version: 2 [ 0.051013] ... bit width: 48 [ 0.052015] ... generic registers: 4 [ 0.053014] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055015] ... fixed-purpose events: 3 [ 0.056014] ... event mask: 000000070000000f [ 0.057413] rcu: Hierarchical SRCU implementation. [ 0.059550] smp: Bringing up secondary CPUs ... [ 0.060725] x86: Booting SMP configuration: [ 0.061025] .... node #0, CPUs: #1 #2 #3 [ 0.064867] smp: Brought up 1 node, 4 CPUs [ 0.066011] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148024] node 0 deferred pages initialised in 78ms [ 0.152110] devtmpfs: initialized [ 0.153294] x86/mm: Memory block size: 128MB [ 0.157457] gcov: version magic: 0x41383552 [ 0.161200] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167263] pinctrl core: initialized pinctrl subsystem [ 0.168184] [ 0.168786] ************************************************************* [ 0.171015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174013] ** ** [ 0.176012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178013] ** ** [ 0.180013] ** This means that this kernel is built to expose internal ** [ 0.183072] ** IOMMU data structures, which may compromise security on ** [ 0.185009] ** your system. ** [ 0.187013] ** ** [ 0.189011] ** If you see this message and you are not debugging the ** [ 0.191012] ** kernel, report this immediately to your vendor! ** [ 0.194013] ** ** [ 0.196010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198016] ************************************************************* [ 0.200721] NET: Registered protocol family 16 [ 0.203482] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.206069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212012] cpuidle: using governor menu [ 0.213747] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215475] PCI: Using configuration type 1 for base access [ 0.217090] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231178] cryptd: max_cpu_qlen set to 1000 [ 0.234199] ACPI: Added _OSI(Module Device) [ 0.239017] ACPI: Added _OSI(Processor Device) [ 0.241016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248195] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255400] ACPI: Interpreter enabled [ 0.257078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.259029] ACPI: Using IOAPIC for interrupt routing [ 0.261135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264454] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.275258] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.283084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.288529] acpiphp: Slot [2] registered [ 0.290116] acpiphp: Slot [5] registered [ 0.292135] acpiphp: Slot [6] registered [ 0.293116] acpiphp: Slot [7] registered [ 0.294112] acpiphp: Slot [8] registered [ 0.296144] acpiphp: Slot [9] registered [ 0.297119] acpiphp: Slot [10] registered [ 0.299191] acpiphp: Slot [3] registered [ 0.301108] acpiphp: Slot [4] registered [ 0.302102] acpiphp: Slot [11] registered [ 0.304100] acpiphp: Slot [12] registered [ 0.305100] acpiphp: Slot [13] registered [ 0.306116] acpiphp: Slot [14] registered [ 0.308111] acpiphp: Slot [15] registered [ 0.309100] acpiphp: Slot [16] registered [ 0.311098] acpiphp: Slot [17] registered [ 0.312100] acpiphp: Slot [18] registered [ 0.314140] acpiphp: Slot [19] registered [ 0.315105] acpiphp: Slot [20] registered [ 0.316088] acpiphp: Slot [21] registered [ 0.318064] acpiphp: Slot [22] registered [ 0.320117] acpiphp: Slot [23] registered [ 0.322118] acpiphp: Slot [24] registered [ 0.324130] acpiphp: Slot [25] registered [ 0.326162] acpiphp: Slot [26] registered [ 0.328116] acpiphp: Slot [27] registered [ 0.330127] acpiphp: Slot [28] registered [ 0.332117] acpiphp: Slot [29] registered [ 0.334139] acpiphp: Slot [30] registered [ 0.336314] acpiphp: Slot [31] registered [ 0.338073] PCI host bridge to bus 0000:00 [ 0.340025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.343027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.346104] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.349030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.357032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360921] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364268] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373754] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.379050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.382023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.384025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389930] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.393895] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.396046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.399637] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.404011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.414930] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.420017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.427000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.446020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.500669] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.514019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.526086] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.557021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.576012] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.586017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.597020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.623023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.640317] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.652017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.667019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.692019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.713526] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.723017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.731017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.746016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.759101] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.776020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.786030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.808031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.820137] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.825382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.829391] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.831393] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.834234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.840029] iommu: Default domain type: Passthrough [ 0.841664] SCSI subsystem initialized [ 0.843100] ACPI: bus type USB registered [ 0.844191] usbcore: registered new interface driver usbfs [ 0.845058] usbcore: registered new interface driver hub [ 0.847082] usbcore: registered new device driver usb [ 0.848175] pps_core: LinuxPPS API ver. 1 registered [ 0.850011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.852051] PTP clock support registered [ 0.854182] EDAC MC: Ver: 3.0.0 [ 0.856115] PCI: Using ACPI for IRQ routing [ 0.856823] NetLabel: Initializing [ 0.858010] NetLabel: domain hash size = 128 [ 0.858874] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.860124] NetLabel: unlabeled traffic allowed by default [ 0.863104] vgaarb: loaded [ 0.864273] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.866021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.872501] clocksource: Switched to clocksource kvm-clock [ 0.976402] VFS: Disk quotas dquot_6.6.0 [ 0.978184] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.980833] *** VALIDATE ramfs *** [ 0.982148] *** VALIDATE hugetlbfs *** [ 0.983666] pnp: PnP ACPI init [ 0.986198] pnp: PnP ACPI: found 6 devices [ 1.001793] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.005038] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.007254] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.009563] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.012010] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.014302] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.017556] NET: Registered protocol family 2 [ 1.020119] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.025073] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.028591] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.034588] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.038836] TCP: Hash tables configured (established 65536 bind 65536) [ 1.042658] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.047285] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050599] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053386] NET: Registered protocol family 1 [ 1.056325] RPC: Registered named UNIX socket transport module. [ 1.058401] RPC: Registered udp transport module. [ 1.059993] RPC: Registered tcp transport module. [ 1.061864] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.064288] NET: Registered protocol family 44 [ 1.066205] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.068369] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.070263] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.072758] PCI: CLS 0 bytes, default 64 [ 1.074720] Unpacking initramfs... [ 2.530876] debug: unmapping init [mem 0xffff9f6fbcc54000-0xffff9f6fbffbffff] [ 2.533956] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.536615] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.539513] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.976951] Initialise system trusted keyrings [ 2.978161] Key type blacklist registered [ 2.979454] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.986445] zbud: loaded [ 2.989665] *** VALIDATE nfs *** [ 2.991019] *** VALIDATE nfs4 *** [ 2.992873] pstore: using deflate compression [ 2.996061] Platform Keyring initialized [ 3.111482] NET: Registered protocol family 38 [ 3.112832] Key type asymmetric registered [ 3.113827] Asymmetric key parser 'x509' registered [ 3.116402] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.118583] io scheduler mq-deadline registered [ 3.119705] io scheduler kyber registered [ 3.120861] io scheduler bfq registered [ 3.122328] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.124361] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.126439] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.128301] ACPI: Power Button [PWRF] [ 3.133078] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.137821] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.146539] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.151531] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.165258] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.190969] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.220164] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.225508] Non-volatile memory driver v1.3 [ 3.227594] Linux agpgart interface v0.103 [ 3.269529] virtio_blk virtio1: [vda] 145784 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.273831] vda: detected capacity change from 0 to 74641408 [ 3.288321] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.291492] vdb: detected capacity change from 0 to 1073741824 [ 3.305647] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.308943] vdc: detected capacity change from 0 to 2621440000 [ 3.324147] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.327487] vdd: detected capacity change from 0 to 2621440000 [ 3.346538] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.349738] vde: detected capacity change from 0 to 4294967296 [ 3.372324] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.375882] vdf: detected capacity change from 0 to 4294967296 [ 3.383708] libphy: Fixed MDIO Bus: probed [ 3.390397] usbcore: registered new interface driver usbserial_generic [ 3.392976] usbserial: USB Serial support registered for generic [ 3.395337] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.399487] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.401358] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.403630] mousedev: PS/2 mouse device common for all mice [ 3.406528] rtc_cmos 00:05: RTC can wake from S4 [ 3.409643] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.414080] rtc_cmos 00:05: registered as rtc0 [ 3.416334] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.417141] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.420718] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.422390] intel_pstate: CPU model not supported [ 3.429129] hid: raw HID events driver (C) Jiri Kosina [ 3.431340] usbcore: registered new interface driver usbhid [ 3.433470] usbhid: USB HID core driver [ 3.435058] drop_monitor: Initializing network drop monitor service [ 3.437685] Initializing XFRM netlink socket [ 3.439988] NET: Registered protocol family 10 [ 3.443399] Segment Routing with IPv6 [ 3.445206] NET: Registered protocol family 17 [ 3.447824] mpls_gso: MPLS GSO support [ 3.453949] RAS: Correctable Errors collector initialized. [ 3.457048] AVX version of gcm_enc/dec engaged. [ 3.459739] AES CTR mode by8 optimization enabled [ 3.540351] sched_clock: Marking stable (3540325744, 0)->(4326570899, -786245155) [ 3.545263] registered taskstats version 1 [ 3.547175] Loading compiled-in X.509 certificates [ 3.549136] zswap: loaded using pool lzo/zbud [ 3.574811] Key type big_key registered [ 3.588844] Key type encrypted registered [ 3.590589] ima: No TPM chip found, activating TPM-bypass! [ 3.592816] ima: Allocated hash algorithm: sha1 [ 3.594599] ima: No architecture policies found [ 3.596466] evm: Initialising EVM extended attributes: [ 3.598433] evm: security.selinux [ 3.599646] evm: security.ima [ 3.600829] evm: security.capability [ 3.602392] evm: HMAC attrs: 0x1 [ 3.605265] rtc_cmos 00:05: setting system clock to 2026-07-13 18:13:15 UTC (1783966395) [ 3.611851] debug: unmapping init [mem 0xffffffffa1203000-0xffffffffa13fffff] [ 3.615097] debug: unmapping init [mem 0xffffffff9ff82000-0xffffffffa0258fff] [ 3.624112] Write protecting the kernel read-only data: 28672k [ 3.627862] debug: unmapping init [mem 0xffffffff9e603000-0xffffffff9e7fffff] [ 3.631227] debug: unmapping init [mem 0xffffffff9ef14000-0xffffffff9effffff] [ 3.671887] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.680449] systemd[1]: Detected virtualization kvm. [ 3.682491] systemd[1]: Detected architecture x86-64. [ 3.684477] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.711540] systemd[1]: No hostname configured. [ 3.712967] systemd[1]: Set hostname to . [ 3.714850] random: systemd: uninitialized urandom read (16 bytes read) [ 3.717500] systemd[1]: Initializing machine ID from random generator. [ 3.759905] random: ln: uninitialized urandom read (6 bytes read) [ 3.853464] random: systemd: uninitialized urandom read (16 bytes read) [ 3.855305] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.859280] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.863992] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.528471] device-mapper: uevent: version 1.0.3 [ 4.530790] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 5.129396] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.340263] virtio_net virtio0 ens2: renamed from eth0 [ 5.369935] scsi host0: ata_piix [ 5.381209] scsi host1: ata_piix [ 5.382723] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.384986] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.884144] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.113255] random: crng init done [ 10.116695] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.667872] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.410965] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.264927] SELinux: Disabled at runtime. [ 17.437231] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.455869] systemd[1]: Detected virtualization kvm. [ 17.459526] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.950528] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.957568] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.966075] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.977746] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.986498] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.016881] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.042270] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ 19.152120] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.251057] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.285493] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.288318] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.640780] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.777210] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 26.279583] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.339897] NFS: Registering the id_resolver key type [ 27.342362] Key type id_resolver registered [ 27.344202] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg214-server login: [ 90.871266] libcfs: loading out-of-tree module taints kernel. [ 90.951641] Key type ._llcrypt registered [ 90.953758] Key type .llcrypt registered [ 91.111956] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 111.270350] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 113.560789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.576867] alg: No test for adler32 (adler32-zlib) [ 115.205183] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 116.342168] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 118.167448] Key type lgssc registered [ 120.565191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.975122] hrtimer: interrupt took 8750049 ns [ 138.444459] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 185.148717] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 198.597316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.628854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.960853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.991585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.085988] Lustre: lustre-MDT0000: new disk, initializing [ 200.210619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.233671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.992776] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.201287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.290429] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.319272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.325795] Lustre: Skipped 1 previous similar message [ 220.440311] Lustre: lustre-MDT0001: new disk, initializing [ 220.515853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.558702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.571672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.461915] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.366770] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.591200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.901653] Lustre: lustre-OST0000: new disk, initializing [ 240.908840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.923651] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.040205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.910290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.919102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.055124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 248.053714] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.684471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 262.781989] Lustre: lustre-OST0001: new disk, initializing [ 262.787535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 262.793283] Lustre: 9527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 262.886983] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.660890] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.939896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.947216] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 271.999333] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 281.973768] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.757497] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.147730] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing check_logdir /tmp/testlogs/ [ 301.423989] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing yml_node [ 305.548500] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 307.977578] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 310.703225] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 312.375069] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 13 14:18:24 EDT 2026 [ 327.773269] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 329.057299] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 348.639796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 348.643995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.663072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.154887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.172028] Lustre: Skipped 1 previous similar message [ 349.175246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.183525] Lustre: Skipped 3 previous similar messages [ 351.219261] Lustre: server umount lustre-MDT0000 complete [ 356.838456] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783966748 with bad export cookie 8784428494964062381 [ 356.844697] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.852439] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 359.392216] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.399362] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.406455] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 359.435561] Lustre: Skipped 1 previous similar message [ 359.443128] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.445737] Lustre: Skipped 1 previous similar message [ 361.442768] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.450396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.470698] Lustre: Skipped 1 previous similar message [ 366.593227] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.601078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.617461] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 366.632861] Lustre: Skipped 1 previous similar message [ 371.168519] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 371.389437] Lustre: server umount lustre-MDT0001 complete [ 375.228302] Lustre: server umount lustre-OST0000 complete [ 378.714632] Lustre: server umount lustre-OST0001 complete [ 384.379851] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 397.341943] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 437.194149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.526893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 437.577263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 437.657512] Lustre: lustre-MDT0000: new disk, initializing [ 437.705118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.718332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 442.016818] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 453.960449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.125255] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 454.197816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 454.201301] Lustre: Skipped 1 previous similar message [ 454.266381] Lustre: lustre-MDT0001: new disk, initializing [ 454.373595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.415181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 454.432795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.234756] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 470.094906] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 476.030035] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 482.124237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.351765] Lustre: lustre-OST0000: new disk, initializing [ 482.354851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 482.360471] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 482.439063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.435733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 484.447632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 484.604545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 488.693352] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 497.043503] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 501.562723] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 501.775180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 506.812614] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.049335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.436121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.442147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.445447] LustreError: Skipped 1 previous similar message [ 510.459551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.466863] Lustre: Skipped 2 previous similar messages [ 510.477745] Lustre: Skipped 1 previous similar message [ 515.409461] Lustre: server umount lustre-OST0000 complete [ 520.678339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.709861] Lustre: Skipped 1 previous similar message [ 525.343486] Lustre: server umount lustre-MDT0000 complete [ 525.797778] LustreError: 15967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.829227] LustreError: 15967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 529.441170] LustreError: 15887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783966921 with bad export cookie 8784428494964063914 [ 529.447395] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.463617] LustreError: 15887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 529.876660] Lustre: server umount lustre-MDT0001 complete [ 534.149900] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:22:06 (1783966926) [ 536.212628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 537.776920] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:22:09 (1783966929) [ 545.038441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.525651] LustreError: 19462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.641803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.765948] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 556.010273] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.031468] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 558.813504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.985825] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 571.454961] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 576.889919] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 584.103349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 591.684812] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 599.193822] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 611.296570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 611.308645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.348030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 611.363235] Lustre: Skipped 2 previous similar messages [ 614.372254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 625.119433] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 625.457792] Lustre: server umount lustre-OST0000 complete [ 629.733232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.739806] Lustre: Skipped 2 previous similar messages [ 635.898334] Lustre: server umount lustre-MDT0000 complete [ 639.976164] LustreError: 20451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.020991] LustreError: 20451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 640.530312] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783967032 with bad export cookie 8784428494964064796 [ 640.544836] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.546462] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 641.037807] Lustre: server umount lustre-MDT0001 complete [ 654.795155] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 658.179907] Key type lgssc unregistered [ 658.470793] LNet: 22995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.487561] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.504655] LNet: Removed LNI 192.168.202.114@tcp [ 659.881860] Key type .llcrypt unregistered [ 659.886746] Key type ._llcrypt unregistered [ 685.495839] Key type ._llcrypt registered [ 685.497550] Key type .llcrypt registered [ 685.697320] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 686.795168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.852071] alg: No test for adler32 (adler32-zlib) [ 688.277095] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 688.559985] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 690.343164] Key type lgssc registered [ 691.855804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 702.321090] loop: module loaded [ 702.441890] loop0: detected capacity change from 0 to 204800000 [ 702.727439] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.752373] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.116311] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.010304] loop0: detected capacity change from 0 to 204800000 [ 709.265897] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.277400] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.290361] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 714.268281] loop0: detected capacity change from 0 to 204800000 [ 719.556411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 719.591273] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.917773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 721.006780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 721.065341] Lustre: lustre-MDT0000: new disk, initializing [ 721.164562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 721.193618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 726.200513] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 734.001058] loop1: detected capacity change from 0 to 204800000 [ 739.638069] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.935425] Lustre: lustre-OST07c6: new disk, initializing [ 739.937940] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 739.941565] Lustre: Skipped 1 previous similar message [ 739.946159] Lustre: 26186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 740.054870] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 745.999764] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 749.708526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 749.716322] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 749.758181] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 751.180966] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 751.189516] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 751.195967] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 751.204640] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 751.210270] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 751.215378] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 766.938462] Lustre: 26193:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 766.957565] Lustre: 26193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 766.965582] Lustre: 26193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.980693] Lustre: 26193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 766.990539] Lustre: 26193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 767.000410] Lustre: 26193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.762806] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 771.770331] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 771.800856] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 771.813996] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.819434] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.824756] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.833894] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 771.843642] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.862933] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 771.874255] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.881802] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.896497] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.089108] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 775.114636] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 775.125388] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 775.135061] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.152059] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.159858] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.166084] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 775.178747] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.186037] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 775.192459] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.203351] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 775.210523] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.922661] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 779.935616] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 779.941756] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 779.951849] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.961216] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 779.978152] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.995639] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 780.005570] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.018152] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 780.029494] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 780.038323] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 780.045670] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.988951] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 790.997620] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 791.003805] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 791.019839] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.024336] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 791.032077] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.039913] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 791.046825] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.053828] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 791.060160] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 791.065534] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 791.072903] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 838.726886] Lustre: server umount lustre-OST07c6 complete [ 860.395676] Lustre: server umount lustre-MDT0000 complete [ 878.872597] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 882.356375] Key type lgssc unregistered [ 882.617075] LNet: 28744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.622800] LNetError: 28744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.690438] LNet: Removed LNI 192.168.202.114@tcp [ 884.775427] Key type .llcrypt unregistered [ 884.781516] Key type ._llcrypt unregistered [ 905.112483] Key type ._llcrypt registered [ 905.115773] Key type .llcrypt registered [ 905.284082] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 906.584817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.863365] alg: No test for adler32 (adler32-zlib) [ 908.053340] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 908.375434] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 910.079194] Key type lgssc registered [ 911.231114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.908352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 918.926364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 920.240095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.791201] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 930.789960] LustreError: 30017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.814673] LustreError: 30017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 933.905864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.251496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.848522] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 946.757070] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.258403] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.826662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.308196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.424841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 967.754320] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 975.769889] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 995.307047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.313482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 995.323112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 995.350835] Lustre: Skipped 1 previous similar message [ 999.194887] Lustre: server umount lustre-OST0000 complete [ 1005.543917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.554651] Lustre: Skipped 1 previous similar message [ 1005.561419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.966819] Lustre: server umount lustre-MDT0000 complete [ 1010.657109] LustreError: 30017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1010.678131] LustreError: 30017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1012.859242] LustreError: 29997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783967404 with bad export cookie 16321942130408247443 [ 1012.870934] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.872980] LustreError: 29997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1013.304191] Lustre: server umount lustre-MDT0001 complete [ 1025.062713] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1027.987505] Key type lgssc unregistered [ 1028.221495] LNet: 33756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.227591] LNetError: 33756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.245983] LNet: Removed LNI 192.168.202.114@tcp [ 1029.319941] Key type .llcrypt unregistered [ 1029.322560] Key type ._llcrypt unregistered [ 1045.119176] Key type ._llcrypt registered [ 1045.120481] Key type .llcrypt registered [ 1045.200279] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1045.889064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1045.901449] alg: No test for adler32 (adler32-zlib) [ 1046.909410] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 1047.091782] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1048.727414] Key type lgssc registered [ 1049.637672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1055.483419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1055.498792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.936945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.811892] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1066.470569] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.479967] LustreError: 35006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1068.032108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.461063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1072.682701] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1079.145398] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1084.597699] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1091.478085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1091.896837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.900214] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1102.146250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1105.605461] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1117.670931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.674544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.710526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.850362] Lustre: server umount lustre-OST0000 complete [ 1127.904801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.911664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.941301] Lustre: Skipped 2 previous similar messages [ 1127.949477] Lustre: Skipped 1 previous similar message [ 1132.011698] Lustre: server umount lustre-MDT0000 complete [ 1133.029511] LustreError: 36057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.054578] LustreError: 36057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1135.525929] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783967527 with bad export cookie 2241346935283893892 [ 1135.538415] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.540681] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1135.998785] Lustre: server umount lustre-MDT0001 complete [ 1147.362477] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1150.482560] Key type lgssc unregistered [ 1150.782801] LNet: 38539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.790218] LNetError: 38539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.823371] LNet: Removed LNI 192.168.202.114@tcp [ 1151.722634] Key type .llcrypt unregistered [ 1151.724916] Key type ._llcrypt unregistered [ 1174.797484] Key type ._llcrypt registered [ 1174.800495] Key type .llcrypt registered [ 1174.900904] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1175.944521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1176.128221] alg: No test for adler32 (adler32-zlib) [ 1177.164896] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 1177.415616] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1179.063187] Key type lgssc registered [ 1180.301675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1188.487136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1188.529626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.005880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.227603] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1199.590463] LustreError: 39963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.617553] LustreError: 39963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1203.474902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.099997] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1208.988187] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1217.674292] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1224.145508] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1230.964581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1231.232510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.314549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1237.816702] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1243.953546] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1255.392535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1255.401885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.427294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.113591] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.14@tcp (stopping) [ 1258.983034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.983260] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.008207] Lustre: Skipped 1 previous similar message [ 1260.666712] Lustre: server umount lustre-MDT0000 complete [ 1262.199750] LustreError: 39959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.105325] LustreError: 39964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.120349] LustreError: 39964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1264.951151] LustreError: 39944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783967656 with bad export cookie 7001675867491632455 [ 1264.958860] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.962571] LustreError: 39944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1267.312906] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.346485] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [ 1267.357540] Lustre: Skipped 2 previous similar messages [ 1269.221024] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.423978] Lustre: server umount lustre-MDT0001 complete [ 1286.565979] Lustre: server umount lustre-OST0000 complete [ 1304.440633] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1307.677316] Key type lgssc unregistered [ 1308.001556] LNet: 43825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.005545] LNetError: 43825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.013961] LNet: Removed LNI 192.168.202.114@tcp [ 1308.813302] Key type .llcrypt unregistered [ 1308.818820] Key type ._llcrypt unregistered [ 1333.435486] Key type ._llcrypt registered [ 1333.438713] Key type .llcrypt registered [ 1333.531553] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1335.179207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.224034] alg: No test for adler32 (adler32-zlib) [ 1336.434375] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 1336.632358] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1338.287250] Key type lgssc registered [ 1339.703371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.526937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.539743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.050412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.411446] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1357.305980] LustreError: 45241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.320092] LustreError: 45241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1361.289195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.659633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1366.386467] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1373.401770] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1378.327207] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1384.661310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.203711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.340481] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1392.388433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1400.202426] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1406.944510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1406.951298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.965748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1407.981587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.990361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1412.777512] Lustre: server umount lustre-OST0000 complete [ 1432.226656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.230164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1432.244847] Lustre: Skipped 1 previous similar message [ 1432.246748] Lustre: Skipped 2 previous similar messages [ 1436.542106] Lustre: server umount lustre-MDT0000 complete [ 1437.669421] LustreError: 45242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.689502] LustreError: 45242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1440.021824] LustreError: 45222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783967831 with bad export cookie 8993044414028678122 [ 1440.034232] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1446.496376] Lustre: server umount lustre-MDT0001 complete [ 1455.729539] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1458.647689] Key type lgssc unregistered [ 1459.027521] LNet: 48952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.035397] LNetError: 48952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.053448] LNet: Removed LNI 192.168.202.114@tcp [ 1460.013876] Key type .llcrypt unregistered [ 1460.016080] Key type ._llcrypt unregistered [ 1482.936538] Key type ._llcrypt registered [ 1482.939721] Key type .llcrypt registered [ 1483.036335] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1484.128349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.300039] alg: No test for adler32 (adler32-zlib) [ 1485.421727] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 1485.642489] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1487.303198] Key type lgssc registered [ 1488.505834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.614857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.629941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.077757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.337966] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1506.274572] LustreError: 50371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.298332] LustreError: 50371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1509.397135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.625421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1513.539401] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1519.847455] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1524.530456] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1530.867128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1531.117281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.122786] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1541.368765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1545.018153] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1551.354562] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1554.740640] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1566.836958] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-1314h39m18s [ 1568.586371] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1571.297230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1571.309898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.329838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1572.388666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1572.395416] Lustre: Skipped 4 previous similar messages [ 1573.523554] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.14@tcp (stopping) [ 1576.649905] Lustre: server umount lustre-MDT0000 complete [ 1578.470492] LustreError: 50371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.633021] LustreError: 53064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.648633] LustreError: 53064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1579.712867] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1584.621993] LustreError: 53064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1584.645548] LustreError: 53064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1585.659892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.757781] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1585.986151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.027770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1587.000431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1589.681232] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1592.339193] Lustre: DEBUG MARKER: Wait for df (36660) ... [ 1593.977873] Lustre: DEBUG MARKER: done [ 1597.411702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1597.425250] Lustre: Skipped 2 previous similar messages [ 1597.432098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1597.441901] Lustre: Skipped 1 previous similar message [ 1602.529763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1602.532839] Lustre: Skipped 1 previous similar message [ 1603.543837] Lustre: server umount lustre-OST0000 complete [ 1607.648416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.653908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1607.674536] Lustre: Skipped 1 previous similar message [ 1612.755336] Lustre: server umount lustre-MDT0000 complete [ 1613.792926] LustreError: 50371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.804264] LustreError: 50371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1615.736217] LustreError: 52997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968007 with bad export cookie 6621296671186965093 [ 1615.745456] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.984242] Lustre: server umount lustre-MDT0001 complete [ 1624.605640] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1627.137616] Key type lgssc unregistered [ 1627.522682] LNet: 55292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.529090] LNetError: 55292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.561522] LNet: Removed LNI 192.168.202.114@tcp [ 1628.287300] Key type .llcrypt unregistered [ 1628.289100] Key type ._llcrypt unregistered [ 1662.851175] Key type ._llcrypt registered [ 1662.854777] Key type .llcrypt registered [ 1662.946415] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1664.276620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1664.308201] alg: No test for adler32 (adler32-zlib) [ 1665.418585] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 1665.680629] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1667.359663] Key type lgssc registered [ 1668.532355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.261133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.290227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.538231] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1677.553452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1677.721932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.437751] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1691.635537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.702306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1691.734757] Lustre: 57499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1691.952216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1697.168301] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1705.434354] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1711.039437] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1718.779813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.028888] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1719.345194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.766803] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1729.730517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1733.220141] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1740.801438] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-1314h42m12s [ 1742.409641] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1749.406553] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1752.007784] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1752.119608] LustreError: 58509:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1772.178828] Lustre: lustre-MDT0000: Client d6d12fde-250e-43ec-9d53-dc5e0b4bb4af (at 192.168.202.14@tcp) reconnecting [ 1792.207126] LustreError: 58509:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1792.273133] Lustre: lustre-MDT0000: Client d6d12fde-250e-43ec-9d53-dc5e0b4bb4af (at 192.168.202.14@tcp) reconnecting [ 1795.122483] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1797.523059] Lustre: DEBUG MARKER: done [ 1806.307292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1806.324420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.367685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1806.819374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1811.940411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1811.954569] Lustre: Skipped 2 previous similar messages [ 1817.064303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.076249] Lustre: Skipped 1 previous similar message [ 1818.080899] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1818.298911] Lustre: server umount lustre-OST0000 complete [ 1822.178382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.193605] Lustre: Skipped 1 previous similar message [ 1822.197386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1822.210901] Lustre: Skipped 1 previous similar message [ 1828.388367] Lustre: server umount lustre-MDT0000 complete [ 1832.071207] LustreError: 57494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968223 with bad export cookie 2179127580670697121 [ 1832.075533] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1832.133422] LustreError: 57494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1832.841217] Lustre: server umount lustre-MDT0001 complete [ 1847.957486] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1852.067883] Key type lgssc unregistered [ 1852.456718] LNet: 61830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.474293] LNetError: 61830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.544514] LNet: Removed LNI 192.168.202.114@tcp [ 1854.649799] Key type .llcrypt unregistered [ 1854.652559] Key type ._llcrypt unregistered [ 1903.230795] Key type ._llcrypt registered [ 1903.234973] Key type .llcrypt registered [ 1903.341756] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1904.748392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.925835] alg: No test for adler32 (adler32-zlib) [ 1906.179283] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 1906.428881] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1908.119226] Key type lgssc registered [ 1909.547143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1913.101223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1913.133197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.486490] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1914.511379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1914.693678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.889255] Lustre: Failing over lustre-MDT0000 [ 1916.266997] Lustre: server umount lustre-MDT0000 complete [ 1922.511123] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:45:14 (1783968314) [ 1929.479714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.212536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.670925] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1945.074036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.250566] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1945.289570] Lustre: 64632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1945.540458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.929167] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1958.314484] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1964.009318] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1974.163457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.493200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1974.783483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.786083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1984.223158] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 1995.382274] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2002.542828] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2012.128658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2012.140490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.163267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.689413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.690374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.334993] Lustre: server umount lustre-MDT0000 complete [ 2019.791508] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968411 with bad export cookie 9246592460077418778 [ 2019.807021] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.814568] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2019.834275] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.848606] Lustre: Skipped 1 previous similar message [ 2019.855071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2019.861025] Lustre: Skipped 2 previous similar messages [ 2020.082231] Lustre: server umount lustre-MDT0001 complete [ 2023.612277] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2027.894041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2036.184502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.639204] LustreError: 66482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.654738] LustreError: 66482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2044.697263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.790443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2049.219882] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2050.022119] LustreError: 66483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.137573] LustreError: 66482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.436127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.833216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.890949] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2063.366283] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2069.165901] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2070.789911] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2077.407592] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2084.321487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2084.337557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.368550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.954555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.964924] Lustre: Skipped 4 previous similar messages [ 2090.678865] Lustre: server umount lustre-MDT0000 complete [ 2094.481710] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968486 with bad export cookie 9246592460077421053 [ 2094.485353] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.512228] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2094.953249] Lustre: server umount lustre-MDT0001 complete [ 2102.033813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2110.431181] Lustre: 63556:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783968486/real 1783968486] req@000000000e6986ca x1870626342112512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783968502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2110.485395] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.508324] Lustre: Skipped 2 previous similar messages [ 2111.991611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2114.527337] Lustre: 63554:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783968490/real 1783968491] req@0000000066f51d08 x1870626342112640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783968506 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.639286] Lustre: 63554:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783968492/real 1783968492] req@00000000803ce4d3 x1870626342112768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783968508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2119.711696] LustreError: 63553:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004bb70771 x1870626342113536/t0(0) o250->MGC192.168.202.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2120.111397] LustreError: 66483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.217733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.291632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2120.655228] Lustre: 63556:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783968496/real 1783968496] req@000000000e98db69 x1870626342113024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783968512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.203621] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2125.320541] Lustre: 63557:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783968501/real 1783968501] req@000000009981247c x1870626342113152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783968517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2126.309805] LustreError: 71084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.331253] LustreError: 71084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2133.798842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.470912] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2139.442361] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2146.303975] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.022093] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.108735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.110016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2160.141157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.150827] Lustre: Skipped 1 previous similar message [ 2160.521267] Lustre: server umount lustre-MDT0000 complete [ 2164.633074] LustreError: 64627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968556 with bad export cookie 9246592460077422432 [ 2164.637464] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2164.645939] LustreError: 64627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2165.071197] Lustre: server umount lustre-MDT0001 complete [ 2173.159264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2177.568787] Lustre: DEBUG MARKER: files compared the same [ 2190.395495] Lustre: server umount lustre-OST0000 complete [ 2209.871090] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 2213.412082] Key type lgssc unregistered [ 2213.729642] LNet: 74360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.739808] LNetError: 74360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.758347] LNet: Removed LNI 192.168.202.114@tcp [ 2215.939911] Key type .llcrypt unregistered [ 2215.944716] Key type ._llcrypt unregistered [ 2243.060862] Key type ._llcrypt registered [ 2243.063492] Key type .llcrypt registered [ 2243.180406] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2244.491255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2244.540544] alg: No test for adler32 (adler32-zlib) [ 2245.625572] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 2245.892988] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2247.596623] Key type lgssc registered [ 2249.193759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2259.482509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2259.525873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.417272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.725338] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug +malloc all [ 2271.713756] LustreError: 75782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.739350] LustreError: 75782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2276.144715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.448055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.135089] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug +malloc all [ 2294.455246] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2301.907946] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2309.610031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2309.834222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.884605] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug +malloc all [ 2326.349635] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2331.387303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2341.864966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.873496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2341.878420] Lustre: Skipped 1 previous similar message [ 2341.885123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.898875] Lustre: Skipped 1 previous similar message [ 2345.580514] Lustre: server umount lustre-OST0000 complete [ 2352.097490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.115231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2352.124331] Lustre: Skipped 1 previous similar message [ 2357.217177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2357.224885] Lustre: Skipped 1 previous similar message [ 2361.313650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2361.323618] Lustre: Skipped 1 previous similar message [ 2364.383174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2364.583862] Lustre: server umount lustre-MDT0000 complete [ 2366.436562] LustreError: 75781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.453502] LustreError: 75781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2369.138746] LustreError: 75763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968761 with bad export cookie 15049054954138579659 [ 2369.146494] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2369.152888] LustreError: 75763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2369.475520] Lustre: server umount lustre-MDT0001 complete [ 2383.571517] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 2387.653414] Key type lgssc unregistered [ 2388.178173] LNet: 79339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.189686] LNetError: 79339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.226324] LNet: Removed LNI 192.168.202.114@tcp [ 2389.448430] Key type .llcrypt unregistered [ 2389.452676] Key type ._llcrypt unregistered [ 2418.109847] Key type ._llcrypt registered [ 2418.111888] Key type .llcrypt registered [ 2418.291962] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2419.470056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.773090] alg: No test for adler32 (adler32-zlib) [ 2420.857505] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 2421.109989] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2422.831229] Key type lgssc registered [ 2424.184763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2434.791841] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2434.824966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2450.465705] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2455.584059] LustreError: 80733:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.114@tcp: failed processing log, type 4: rc = -110 [ 2482.335347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.597301] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2497.995622] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2506.602627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.804208] Lustre: *** cfs_fail_loc=706, val=0*** [ 2507.375719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.463127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2512.424912] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2512.867026] LustreError: 80759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.879322] LustreError: 80759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2516.975573] LustreError: 81820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.004877] LustreError: 81820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2521.074214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.446096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2525.922626] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2534.652341] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2541.145477] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2547.173172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.183599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2547.200315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2547.225977] Lustre: Skipped 1 previous similar message [ 2549.353956] Lustre: server umount lustre-OST0000 complete [ 2557.422640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2557.430921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.434084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2557.451790] Lustre: Skipped 2 previous similar messages [ 2562.529415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.544954] Lustre: Skipped 2 previous similar messages [ 2567.650466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.662729] Lustre: Skipped 1 previous similar message [ 2568.159164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2568.375468] Lustre: server umount lustre-MDT0000 complete [ 2572.441685] LustreError: 80739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783968964 with bad export cookie 11859069247198860063 [ 2572.448556] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.451149] LustreError: 80739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2572.770838] Lustre: server umount lustre-MDT0001 complete [ 2585.187912] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 2588.246720] Key type lgssc unregistered [ 2588.572132] LNet: 84354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2588.577305] LNetError: 84354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2588.611035] LNet: Removed LNI 192.168.202.114@tcp [ 2589.376429] Key type .llcrypt unregistered [ 2589.377875] Key type ._llcrypt unregistered [ 2619.455969] Key type ._llcrypt registered [ 2619.457863] Key type .llcrypt registered [ 2619.599162] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2621.590340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.617540] alg: No test for adler32 (adler32-zlib) [ 2622.823646] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 2623.092322] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2624.767818] Key type lgssc registered [ 2625.807832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2635.207311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2635.224499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.302304] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2649.888610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.549856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2655.554722] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2655.714985] LustreError: 86459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.750512] LustreError: 86459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2660.832868] LustreError: 86462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.864812] LustreError: 86462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2665.681149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2665.953272] LustreError: 86463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.975447] LustreError: 86463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2666.255333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.593747] LustreError: 86462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.614972] LustreError: 86462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2673.675751] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2675.712698] LustreError: 86463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.740718] LustreError: 86463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2683.420146] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.444687] Lustre: MGS: Client a3b01a57-3d57-428d-ab6f-0de7d3f30d57 (at 0@lo) reconnecting [ 2683.463693] LustreError: 86473:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2683.860655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2683.965785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2689.468845] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2704.353575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.369550] Lustre: Skipped 1 previous similar message [ 2704.372244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2709.473396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2709.486469] Lustre: Skipped 2 previous similar messages [ 2710.775631] Lustre: server umount lustre-OST0000 complete [ 2719.718049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.718942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2719.749141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.115454] Lustre: server umount lustre-MDT0000 complete [ 2724.835952] LustreError: 86462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.852749] LustreError: 86462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2726.352387] Lustre: server umount lustre-MDT0001 complete [ 2730.490474] Lustre: server umount MGS complete [ 2743.534970] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:58:55 (1783969135) [ 2778.829361] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 2794.896787] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2838.059897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.740742] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2858.342068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.814851] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2858.885107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2859.037525] Lustre: lustre-MDT0001: new disk, initializing [ 2859.157206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2859.221829] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2859.241521] Lustre: 93672:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2859.258728] LustreError: 93672:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2865.548609] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2879.229097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2879.611407] Lustre: lustre-OST0000: new disk, initializing [ 2879.617468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2879.623078] Lustre: 94451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2879.736472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.264677] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2890.345618] Lustre: 94835:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007f93dbfc x1870627092866560/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783969298 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2899.778192] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2907.802872] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2907.815516] Lustre: MGS: Client 0f7528a6-bd8b-4dcc-8ea1-8725214a0585 (at 0@lo) reconnecting [ 2907.896405] Lustre: 95497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2907.932090] LustreError: 93658:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2907.947445] LustreError: 93658:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2907.961124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2908.074250] Lustre: lustre-MDT0000: new disk, initializing [ 2908.387952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.425837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2908.453828] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2909.536389] Lustre: 95521:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000c0a5f375 x1870627092873472/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783969317 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2913.808953] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 2914.618978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2914.634897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2914.760227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2940.386611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.390604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.392762] Lustre: Skipped 1 previous similar message [ 2940.398765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.440503] Lustre: Skipped 2 previous similar messages [ 2945.510857] LustreError: 94438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.529977] LustreError: 94438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2945.829782] Lustre: server umount lustre-OST0000 complete [ 2955.749207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2955.753430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.764393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.791040] Lustre: Skipped 2 previous similar messages [ 2955.823057] Lustre: Skipped 1 previous similar message [ 2957.225066] Lustre: server umount lustre-MDT0000 complete [ 2967.861269] Lustre: server umount lustre-MDT0001 complete [ 2972.492399] Lustre: server umount MGS complete [ 2986.432723] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:02:58 (1783969378) [ 3007.254264] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3010.550768] Key type lgssc unregistered [ 3010.851796] LNet: 98017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3010.857984] LNetError: 98017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3010.880504] LNet: Removed LNI 192.168.202.114@tcp [ 3011.797483] Key type .llcrypt unregistered [ 3011.798771] Key type ._llcrypt unregistered [ 3029.823797] Key type ._llcrypt registered [ 3029.825802] Key type .llcrypt registered [ 3029.936943] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3031.497624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3031.521540] alg: No test for adler32 (adler32-zlib) [ 3032.856448] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 3033.094564] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3034.759242] Key type lgssc registered [ 3036.775842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3047.392680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3047.475249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.526692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.383125] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3059.693400] LustreError: 99243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.725395] LustreError: 99243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3063.289555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.505392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3069.450816] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3078.170456] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3087.167671] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3095.253491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.579803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.886826] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3111.516230] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3119.144426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3134.432933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.439297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.453706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3134.461659] Lustre: Skipped 1 previous similar message [ 3135.716256] Lustre: server umount lustre-OST0000 complete [ 3140.579131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3140.591770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.616489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.628988] Lustre: Skipped 1 previous similar message [ 3144.674400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.677334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.707193] Lustre: Skipped 1 previous similar message [ 3149.793880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3154.401775] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3154.583191] Lustre: server umount lustre-MDT0000 complete [ 3154.913338] LustreError: 99243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.918546] LustreError: 99243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3159.665143] LustreError: 99978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969551 with bad export cookie 14257817237581201736 [ 3159.673791] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3159.677847] LustreError: 99978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3160.153871] Lustre: server umount lustre-MDT0001 complete [ 3180.956287] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3184.139518] Key type lgssc unregistered [ 3184.710888] LNet: 102826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3184.733686] LNetError: 102826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3184.774618] LNet: Removed LNI 192.168.202.114@tcp [ 3185.779546] Key type .llcrypt unregistered [ 3185.782226] Key type ._llcrypt unregistered [ 3205.810951] Key type ._llcrypt registered [ 3205.813603] Key type .llcrypt registered [ 3205.965311] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3207.324112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3207.386194] alg: No test for adler32 (adler32-zlib) [ 3208.545134] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 3208.947623] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3210.655268] Key type lgssc registered [ 3211.870977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3218.545452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3218.552438] LustreError: 103907:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3218.571691] LustreError: 103910:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3218.578348] LustreError: 103907:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3218.582150] LustreError: 103907:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3218.589121] LustreError: 103907:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3218.638834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.165165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.264529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.649528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3233.048054] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3239.069493] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing lsmod [ 3242.344990] LustreError: 105106:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3242.423044] LustreError: 105115:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3242.431186] LustreError: 105106:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4924 [ 3242.436939] LustreError: 105106:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3242.448467] LustreError: 105106:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3242.455079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.831328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.944084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3246.046583] LustreError: 103967:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 3249.124080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3249.134314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.162221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3251.168676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3251.172687] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3251.179502] Lustre: Skipped 1 previous similar message [ 3251.211043] Lustre: Skipped 1 previous similar message [ 3253.293830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.298538] Lustre: Skipped 1 previous similar message [ 3258.345815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3258.351435] Lustre: Skipped 1 previous similar message [ 3260.383088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3260.712468] Lustre: server umount lustre-MDT0000 complete [ 3263.456702] LustreError: 103947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.481621] LustreError: 103947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3265.401117] LustreError: 103921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969657 with bad export cookie 13382973493693522035 [ 3265.405715] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3265.413201] LustreError: 103921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3265.588216] LustreError: 104378:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c0a5f375 x1870627707312384/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3265.883650] Lustre: server umount lustre-MDT0001 complete [ 3280.656880] Lustre: server umount lustre-OST0000 complete [ 3290.836391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.341366] LustreError: 106182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.426677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.425752] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3296.738196] LustreError: 106183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.857387] LustreError: 106182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.936421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.394937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3311.492547] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3318.575940] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3326.069058] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3334.575725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3334.959634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.019955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3341.897312] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3350.938845] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3361.760141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.769222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.779678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3361.818814] Lustre: Skipped 1 previous similar message [ 3365.050514] Lustre: server umount lustre-OST0000 complete [ 3372.002055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.006308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3372.019727] Lustre: Skipped 1 previous similar message [ 3372.022351] Lustre: Skipped 1 previous similar message [ 3383.263210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3383.456317] Lustre: server umount lustre-MDT0000 complete [ 3386.348771] LustreError: 107168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.374316] LustreError: 107168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3387.395486] LustreError: 108026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969779 with bad export cookie 13382973493693522952 [ 3387.401211] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3387.410047] LustreError: 108026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3387.838937] Lustre: server umount lustre-MDT0001 complete [ 3398.858735] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3402.561495] Key type lgssc unregistered [ 3403.044109] LNet: 109715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3403.053222] LNetError: 109715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3404.074981] LNet: Removed LNI 192.168.202.114@tcp [ 3405.108213] Key type .llcrypt unregistered [ 3405.109975] Key type ._llcrypt unregistered [ 3431.974686] Key type ._llcrypt registered [ 3431.993386] Key type .llcrypt registered [ 3432.210522] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3433.629957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3433.709709] alg: No test for adler32 (adler32-zlib) [ 3434.956331] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 3435.338868] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3437.088848] Key type lgssc registered [ 3438.419686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3447.335436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3447.352655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.977473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.420632] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3458.532299] LustreError: 111136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3458.560865] LustreError: 111136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3463.054307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.402419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3467.962548] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3476.215213] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3482.679710] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3490.188084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.695374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.761395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3498.103775] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3506.971697] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3515.376578] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3518.623471] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3524.583127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.588559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3524.590408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3524.590656] Lustre: Skipped 1 previous similar message [ 3526.476812] Lustre: server umount lustre-OST0000 complete [ 3534.819103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3534.834698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3534.834707] Lustre: Skipped 1 previous similar message [ 3534.854243] Lustre: Skipped 1 previous similar message [ 3537.148101] Lustre: server umount lustre-MDT0000 complete [ 3539.938419] LustreError: 111132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.949190] LustreError: 111132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3541.308396] LustreError: 111115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783969933 with bad export cookie 15405414251820425454 [ 3541.331916] LustreError: 111115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3541.333150] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3542.097935] Lustre: server umount lustre-MDT0001 complete [ 3554.081996] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3557.013452] Key type lgssc unregistered [ 3557.290397] LNet: 114777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3557.308288] LNetError: 114777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3557.337063] LNet: Removed LNI 192.168.202.114@tcp [ 3558.629567] Key type .llcrypt unregistered [ 3558.632445] Key type ._llcrypt unregistered [ 3578.889622] Key type ._llcrypt registered [ 3578.894080] Key type .llcrypt registered [ 3579.057731] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3580.646782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3580.864876] alg: No test for adler32 (adler32-zlib) [ 3581.929158] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 3582.131495] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3583.791344] Key type lgssc registered [ 3584.834617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3592.669720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.692925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.342952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.214848] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3604.970307] LustreError: 115999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.024824] LustreError: 115999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3610.089685] LustreError: 115998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3611.470372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.890160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3617.454322] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3625.649217] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3632.350366] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3639.976118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3640.446415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3648.021816] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3657.113690] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3665.028592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3677.667732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3677.676553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.694983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.743461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.744684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.473074] Lustre: server umount lustre-OST0000 complete [ 3688.932291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3688.941708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.954527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3688.967495] Lustre: Skipped 1 previous similar message [ 3690.976985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.978801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.906796] Lustre: server umount lustre-MDT0000 complete [ 3696.101421] LustreError: 115994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3696.117234] LustreError: 115994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3697.250933] LustreError: 115978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970089 with bad export cookie 17613966734512055631 [ 3697.257713] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.285821] LustreError: 115978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3697.830335] Lustre: server umount lustre-MDT0001 complete [ 3710.673509] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3713.829041] Key type lgssc unregistered [ 3714.128885] LNet: 119555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3714.135759] LNetError: 119555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3714.152137] LNet: Removed LNI 192.168.202.114@tcp [ 3715.307398] Key type .llcrypt unregistered [ 3715.310630] Key type ._llcrypt unregistered [ 3748.703972] Key type ._llcrypt registered [ 3748.716633] Key type .llcrypt registered [ 3748.919827] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3750.234934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3750.292812] alg: No test for adler32 (adler32-zlib) [ 3751.777352] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 3752.116426] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3753.823312] Key type lgssc registered [ 3755.506814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3764.893468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3764.913869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.468920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.105311] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3776.995480] LustreError: 120975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3777.025210] LustreError: 120975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3782.158975] LustreError: 120974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3782.232278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.736717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3788.567616] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3798.364301] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3804.348936] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3813.358555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3813.687210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3820.382286] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3828.114130] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3836.214866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3848.617620] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3854.123772] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3855.119994] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3855.124810] Lustre: Skipped 2 previous similar messages [ 3860.777756] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3860.781130] Lustre: Skipped 2 previous similar messages [ 3863.723092] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3863.793888] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3863.797955] Lustre: Skipped 1 previous similar message [ 3869.880486] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.14@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3869.906333] Lustre: Skipped 2 previous similar messages [ 3873.947786] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.14@tcp 192.168.0.[2,10]@tcp [ 3879.442433] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.14/24@tcp in log lustre-MDT0000 [ 3879.451529] Lustre: Skipped 2 previous similar messages [ 3881.857642] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.14/24@tcp [ 3881.865105] Lustre: Skipped 1 previous similar message [ 3894.754079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3894.761675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.782869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.831682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.832054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.862810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3899.082785] Lustre: server umount lustre-OST0000 complete [ 3905.504578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3905.513985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.532172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3905.534915] Lustre: Skipped 1 previous similar message [ 3908.066449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3908.067443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.186891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3913.192028] Lustre: Skipped 2 previous similar messages [ 3917.280361] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3917.583907] Lustre: server umount lustre-MDT0000 complete [ 3918.306101] LustreError: 120975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.317079] LustreError: 120975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3921.752915] LustreError: 120955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970313 with bad export cookie 4742701966504044604 [ 3921.760067] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3921.766152] LustreError: 120955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3922.460938] Lustre: server umount lustre-MDT0001 complete [ 3934.907171] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3938.082217] Key type lgssc unregistered [ 3938.457655] LNet: 125809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3938.464218] LNetError: 125809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3938.482615] LNet: Removed LNI 192.168.202.114@tcp [ 3939.598174] Key type .llcrypt unregistered [ 3939.599523] Key type ._llcrypt unregistered [ 3968.526935] Key type ._llcrypt registered [ 3968.529953] Key type .llcrypt registered [ 3968.665666] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3969.869407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3969.946200] alg: No test for adler32 (adler32-zlib) [ 3970.965565] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 3971.194479] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3972.929090] Key type lgssc registered [ 3974.183985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3981.717285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3981.729632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.143257] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3983.172835] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.14/24@tcp [ 3983.184518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.120200] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 3993.575416] LustreError: 127371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.601897] LustreError: 127371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3997.962287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3998.298396] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3998.336289] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.14/24@tcp [ 3998.404646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4003.777371] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4011.459320] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4018.931811] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4025.900364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4026.257071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4028.267664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4032.464766] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4043.062291] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4057.476484] Lustre: server umount lustre-OST0000 complete [ 4064.229727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.237308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4064.253442] Lustre: Skipped 1 previous similar message [ 4064.266056] Lustre: Skipped 1 previous similar message [ 4069.345655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4069.351569] Lustre: Skipped 1 previous similar message [ 4071.502742] Lustre: server umount lustre-MDT0000 complete [ 4074.465338] LustreError: 128587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.478668] LustreError: 128587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4075.489085] Lustre: 126684:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783970451/real 1783970451] req@00000000aafe636c x1870628507384192/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783970467 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4075.495944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4075.518178] Lustre: 126684:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4075.555891] LustreError: 127371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.584759] LustreError: 127371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4075.609299] LustreError: 127353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970467 with bad export cookie 10118887760998588836 [ 4075.626099] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.181955] Lustre: server umount lustre-MDT0001 complete [ 4088.826423] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 4092.283647] Key type lgssc unregistered [ 4092.560511] LNet: 130960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4092.566315] LNetError: 130960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4092.579201] LNet: Removed LNI 192.168.202.114@tcp [ 4093.510756] Key type .llcrypt unregistered [ 4093.513531] Key type ._llcrypt unregistered [ 4122.656779] Key type ._llcrypt registered [ 4122.660158] Key type .llcrypt registered [ 4122.777325] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 4139.035511] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4140.275942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4140.345493] alg: No test for adler32 (adler32-zlib) [ 4141.501413] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 4141.813389] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 4143.647226] Key type lgssc registered [ 4145.096876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4186.027807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4186.112792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.482681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4187.551859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4187.700975] Lustre: lustre-MDT0000: new disk, initializing [ 4187.858759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.901615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4193.832856] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4205.548378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4205.694310] Lustre: 134363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4205.743218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4205.748120] Lustre: Skipped 1 previous similar message [ 4205.829673] Lustre: lustre-MDT0001: new disk, initializing [ 4205.985719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4206.071242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4206.090547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4211.907914] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4222.671396] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4229.329929] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4236.506086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4236.859428] Lustre: lustre-OST0000: new disk, initializing [ 4236.865051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4236.882465] Lustre: 136333:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4236.989329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.032133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4243.050307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4243.236212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4245.355663] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4256.795414] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4262.065840] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4262.301273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4267.491501] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4267.795575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4274.167671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.184150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.204143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.250678] LustreError: Skipped 1 previous similar message [ 4276.138903] Lustre: server umount lustre-OST0000 complete [ 4282.853315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4282.873322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.892458] Lustre: Skipped 1 previous similar message [ 4282.905609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.913047] Lustre: Skipped 1 previous similar message [ 4284.387993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.397789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.673366] Lustre: server umount lustre-MDT0000 complete [ 4290.752592] LustreError: 136332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970682 with bad export cookie 13896695816375509609 [ 4290.757778] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4290.763427] LustreError: 136332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4291.020823] Lustre: server umount lustre-MDT0001 complete [ 4300.073662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.795714] LustreError: 138549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4300.828614] LustreError: 138549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4300.929298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4306.190201] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4306.407514] LustreError: 138550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4310.499273] LustreError: 138549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.622972] LustreError: 138550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.936825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.442373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4321.526562] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4330.303574] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4336.525290] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4343.228561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4343.727074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.756106] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4359.708580] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4365.183138] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4365.406751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4370.289577] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4370.627690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4379.504513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4379.712563] Lustre: lustre-OST0001: new disk, initializing [ 4379.714977] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4379.721708] Lustre: 141919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4379.834873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4386.662979] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4386.962274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4386.992859] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4387.137829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4394.828185] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4395.054310] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4400.171646] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4400.479911] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4407.639736] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88b3987bf000.ost_server_uuid 50 [ 4409.727063] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88b3987bf000.ost_server_uuid in IDLE state after 0 sec [ 4420.177567] Lustre: server umount lustre-OST0001 complete [ 4423.136851] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4423.139575] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.144052] LustreError: 142242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.166580] Lustre: Skipped 1 previous similar message [ 4427.746170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4427.754747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.759153] LustreError: Skipped 2 previous similar messages [ 4427.762733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4427.768194] Lustre: Skipped 1 previous similar message [ 4427.790789] Lustre: Skipped 2 previous similar messages [ 4430.773295] Lustre: server umount lustre-OST0000 complete [ 4438.498198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.526613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4438.532632] Lustre: Skipped 3 previous similar messages [ 4441.773312] Lustre: server umount lustre-MDT0000 complete [ 4443.631647] LustreError: 138549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.659328] LustreError: 138549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4445.892436] LustreError: 141918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783970837 with bad export cookie 13896695816375510491 [ 4445.904055] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4446.512696] Lustre: server umount lustre-MDT0001 complete [ 4467.440292] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 4471.466870] Key type lgssc unregistered [ 4471.893621] LNet: 144922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4471.908496] LNetError: 144922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4471.934906] LNet: Removed LNI 192.168.202.114@tcp [ 4473.175241] Key type .llcrypt unregistered [ 4473.179250] Key type ._llcrypt unregistered [ 4519.382527] Key type ._llcrypt registered [ 4519.385027] Key type .llcrypt registered [ 4519.494431] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 4536.845632] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4537.900150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4538.043043] alg: No test for adler32 (adler32-zlib) [ 4539.295951] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 4539.598109] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 4541.318996] Key type lgssc registered [ 4542.796635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4577.272253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4577.303210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.536153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4578.574589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4578.697860] Lustre: lustre-MDT0000: new disk, initializing [ 4578.772607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.784307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4582.746604] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4595.103311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.293095] Lustre: 149081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4595.444178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4595.452380] Lustre: Skipped 1 previous similar message [ 4595.638470] Lustre: lustre-MDT0001: new disk, initializing [ 4595.828809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4595.852396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4595.863242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4601.301563] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4611.288567] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4616.997891] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4624.019856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4624.401495] Lustre: lustre-OST0000: new disk, initializing [ 4624.404847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4624.418570] Lustre: 151049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4624.518082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4631.552632] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4634.177075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4634.188730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4634.391251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4641.509266] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4645.986899] Lustre: Failing over lustre-OST0000 [ 4646.149944] Lustre: server umount lustre-OST0000 complete [ 4663.265148] Lustre: 147307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783971039/real 1783971039] req@000000007b7e8207 x1870629102978176/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1783971055 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4663.314841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.375231] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783971041/real 1783971041] req@0000000082572657 x1870629102979328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783971057 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4665.378986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.404056] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4665.674831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4666.112076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.131825] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4667.108075] Lustre: 147306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783971041/real 1783971041] req@00000000763a79ec x1870629102979072/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783971057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4670.177712] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783971046/real 1783971046] req@0000000021aa4a72 x1870629102980480/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783971062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4670.216472] Lustre: 147305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4673.268490] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4673.433133] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4675.398498] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4675.401050] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4676.388247] Lustre: 147304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783971052/real 1783971052] req@000000002a5e8e8a x1870629102981504/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783971068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4677.361547] Lustre: Failing over lustre-MDT0000 [ 4677.604308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4677.619479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4677.722272] Lustre: server umount lustre-MDT0000 complete [ 4683.754072] LustreError: 149091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.771285] LustreError: 149091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4684.405547] LustreError: 150215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.865637] LustreError: 149086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.899498] LustreError: 149086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4693.988527] LustreError: 149087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.019678] LustreError: 149087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4695.008461] Lustre: 147304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783971070/real 1783971070] req@000000000ad097a8 x1870629102994048/t0(0) o400->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783971086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4695.038682] Lustre: 147304:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4695.053624] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4697.333876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.762277] LustreError: 149086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4699.785567] LustreError: 149086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4704.224566] LustreError: 147303:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006918391c x1870629102999424/t0(0) o250->MGC192.168.202.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4704.579846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4704.635311] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4704.880217] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4709.263572] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4709.886546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4709.895359] Lustre: Skipped 1 previous similar message [ 4709.940531] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4710.008088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4720.098550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.123531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4720.130140] Lustre: Skipped 3 previous similar messages [ 4720.149509] Lustre: Skipped 1 previous similar message [ 4725.125579] Lustre: server umount lustre-OST0000 complete [ 4730.346049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.354095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4730.354244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4730.360271] Lustre: Skipped 1 previous similar message [ 4735.457091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4735.461510] Lustre: Skipped 2 previous similar messages [ 4740.578100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4740.583977] Lustre: Skipped 1 previous similar message [ 4744.159252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4744.397683] Lustre: server umount lustre-MDT0000 complete [ 4745.697182] LustreError: 149086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.719621] LustreError: 149086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4748.472172] LustreError: 152302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971140 with bad export cookie 14855906832404155309 [ 4748.475710] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4748.484932] LustreError: 152302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4748.956192] Lustre: server umount lustre-MDT0001 complete [ 4762.582934] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 4765.771251] Key type lgssc unregistered [ 4766.080377] LNet: 154706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4766.090332] LNetError: 154706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4766.104296] LNet: Removed LNI 192.168.202.114@tcp [ 4767.056643] Key type .llcrypt unregistered [ 4767.059362] Key type ._llcrypt unregistered [ 4807.569364] Key type ._llcrypt registered [ 4807.572397] Key type .llcrypt registered [ 4807.698397] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4809.739318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4809.784231] alg: No test for adler32 (adler32-zlib) [ 4810.798805] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 4811.009792] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 4812.791779] Key type lgssc registered [ 4813.905744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4825.390511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4825.435267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4827.473593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4833.579769] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4837.861655] LustreError: 156228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.891794] LustreError: 156228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4838.883179] LustreError: 156229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.044234] LustreError: 156228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.903282] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4849.120133] LustreError: 156229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.662416] LustreError: 156257:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4849.676513] LustreError: 156257:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4849.697639] LustreError: 156257:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4850.059094] Lustre: server umount lustre-MDT0000 complete [ 4868.149565] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 4881.264336] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4924.234452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4924.554593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4924.625351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4924.733682] Lustre: lustre-MDT0000: new disk, initializing [ 4924.898360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4924.932528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4930.266930] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4942.319717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4942.444766] Lustre: 159561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4942.503864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4942.511337] Lustre: Skipped 1 previous similar message [ 4942.635397] Lustre: lustre-MDT0001: new disk, initializing [ 4942.767565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4942.822689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4942.840907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4948.735619] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4960.141123] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4967.091404] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4973.961035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.312371] Lustre: lustre-OST0000: new disk, initializing [ 4974.317310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4974.325413] Lustre: 161532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4974.440107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.376872] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 4982.347483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4982.360675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4982.518737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4991.730956] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4996.597508] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4998.325668] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:36:30 (1783971390) [ 5223.392229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.393766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5223.405263] Lustre: Skipped 1 previous similar message [ 5223.417031] Lustre: Skipped 2 previous similar messages [ 5228.020141] Lustre: server umount lustre-MDT0000 complete [ 5228.511851] LustreError: 159575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.550416] LustreError: 159575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5232.048569] LustreError: 159555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971623 with bad export cookie 8592470342536554354 [ 5232.050692] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.057360] LustreError: 159555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5233.632056] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.645527] Lustre: Skipped 1 previous similar message [ 5233.649863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5237.663832] LustreError: 159569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.664347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5237.686776] LustreError: 159569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5238.477194] Lustre: server umount lustre-MDT0001 complete [ 5248.833821] Lustre: server umount lustre-OST0000 complete [ 5260.058347] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 5271.622383] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5318.864618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5319.429680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5319.491244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5319.696146] Lustre: lustre-MDT0000: new disk, initializing [ 5319.871834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5319.929410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5326.408279] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5338.372930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.590643] Lustre: 166190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5338.743581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5338.747278] Lustre: Skipped 1 previous similar message [ 5338.899147] Lustre: lustre-MDT0001: new disk, initializing [ 5339.026254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5339.081819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5339.088874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5343.912935] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5354.337154] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5361.329949] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5368.557146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5368.891693] Lustre: lustre-OST0000: new disk, initializing [ 5368.895931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5368.906598] Lustre: 168160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5369.012630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.739609] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5378.116843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5378.127762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5378.226145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5386.070960] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5391.690337] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5391.982497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5397.546534] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5397.795694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5404.145761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.157490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.178574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5409.259928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5409.267214] Lustre: Skipped 1 previous similar message [ 5414.367174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5414.505456] Lustre: server umount lustre-OST0000 complete [ 5419.492324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.500920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5419.511387] Lustre: Skipped 2 previous similar messages [ 5419.524050] Lustre: Skipped 4 previous similar messages [ 5425.059938] Lustre: server umount lustre-MDT0000 complete [ 5429.531328] LustreError: 166183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783971821 with bad export cookie 8592470342536713177 [ 5429.548684] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5430.142733] Lustre: server umount lustre-MDT0001 complete [ 5441.415882] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:43:52 (1783971832) [ 5463.062871] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 5475.620698] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5517.104314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5517.388035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5517.420670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5517.527922] Lustre: lustre-MDT0000: new disk, initializing [ 5517.666977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.697989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5523.007954] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5536.311973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.549867] Lustre: 172921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5536.965227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5542.959301] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5555.018739] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5562.068311] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5569.101656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5569.425167] Lustre: lustre-OST0000: new disk, initializing [ 5569.427136] Lustre: Skipped 1 previous similar message [ 5569.434195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5569.438366] Lustre: Skipped 2 previous similar messages [ 5569.442938] Lustre: 174903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5575.888913] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5578.818868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5578.831121] Lustre: Skipped 1 previous similar message [ 5578.843081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5578.923392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5588.401914] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5599.720335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5599.721991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.735197] LustreError: Skipped 2 previous similar messages [ 5599.754368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5599.756397] Lustre: Skipped 1 previous similar message [ 5599.773454] Lustre: Skipped 3 previous similar messages [ 5602.441885] Lustre: server umount lustre-OST0000 complete [ 5608.929651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5608.933555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.833890] Lustre: server umount lustre-MDT0000 complete [ 5615.074296] LustreError: 172928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.114941] LustreError: 172928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5618.281732] LustreError: 172915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972010 with bad export cookie 8592470342536714059 [ 5618.289818] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5618.300416] LustreError: 172915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5618.958948] Lustre: server umount lustre-MDT0001 complete [ 5631.997421] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:47:03 (1783972023) [ 5650.129066] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 5661.938461] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5699.517843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.687441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5699.719538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5699.808957] Lustre: lustre-MDT0000: new disk, initializing [ 5699.945477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.954888] Lustre: Skipped 2 previous similar messages [ 5699.983571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5704.716434] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5715.970092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.121423] Lustre: 179228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5716.522357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5721.992703] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5731.915891] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5738.261964] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5745.195321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.662696] Lustre: 181203:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5751.769548] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5752.945652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5753.094481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5762.722178] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5773.793593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5773.799748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.808367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5773.816641] Lustre: Skipped 2 previous similar messages [ 5773.833813] Lustre: Skipped 4 previous similar messages [ 5777.708192] Lustre: server umount lustre-OST0000 complete [ 5783.009794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5796.320305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5796.592657] Lustre: server umount lustre-MDT0000 complete [ 5798.376388] LustreError: 182019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.398229] LustreError: 182019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5800.144613] LustreError: 179220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972192 with bad export cookie 8592470342536715151 [ 5800.150385] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5800.164016] LustreError: 179220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5800.481193] Lustre: server umount lustre-MDT0001 complete [ 5812.648742] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 5816.449628] Key type lgssc unregistered [ 5816.831578] LNet: 183060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5816.846700] LNetError: 183060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5816.872099] LNet: Removed LNI 192.168.202.114@tcp [ 5817.821199] Key type .llcrypt unregistered [ 5817.822968] Key type ._llcrypt unregistered [ 5843.833078] Key type ._llcrypt registered [ 5843.839144] Key type .llcrypt registered [ 5843.974748] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5845.560747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5845.690210] alg: No test for adler32 (adler32-zlib) [ 5846.807223] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 5847.188694] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 5848.983218] Key type lgssc registered [ 5850.246908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5859.621695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5859.648980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.119976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5866.347963] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5870.570097] LustreError: 184479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.612275] LustreError: 184479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5875.683864] LustreError: 184480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.397649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5876.769041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5882.567677] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5890.990830] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5897.600162] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5905.071652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5905.508458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.516314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5912.292452] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 5920.977361] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5941.216631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5941.230443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.260733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5942.242613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5942.251316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.290857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5945.312494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5947.523503] Lustre: server umount lustre-OST0000 complete [ 5953.504543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5953.515587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5953.538709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5953.547631] Lustre: Skipped 1 previous similar message [ 5955.554564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.674473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5960.687280] Lustre: Skipped 3 previous similar messages [ 5966.303194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5966.517759] Lustre: server umount lustre-MDT0000 complete [ 5970.857347] LustreError: 184462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972362 with bad export cookie 3853465977552221631 [ 5970.861657] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5970.867914] LustreError: 184462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5971.284504] Lustre: server umount lustre-MDT0001 complete [ 5985.479285] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 5988.722545] Key type lgssc unregistered [ 5989.262594] LNet: 188037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5989.274120] LNetError: 188037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5989.302221] LNet: Removed LNI 192.168.202.114@tcp [ 5990.313644] Key type .llcrypt unregistered [ 5990.321357] Key type ._llcrypt unregistered [ 6018.481710] Key type ._llcrypt registered [ 6018.483435] Key type .llcrypt registered [ 6018.556586] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6019.644927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6019.685685] alg: No test for adler32 (adler32-zlib) [ 6020.870623] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 6021.098667] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 6022.791396] Key type lgssc registered [ 6024.072996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6034.020428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6034.040157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.646562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.469450] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6045.164769] LustreError: 189456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.200300] LustreError: 189456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6050.273638] LustreError: 189455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.567582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.113873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6059.505091] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6068.954853] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6075.830196] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6083.990104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6084.569345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6092.750954] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6101.567882] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6106.098312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6116.325064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.344812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6118.560197] Lustre: lustre-OST0000: Not available for connect from 192.168.202.14@tcp (stopping) [ 6118.571925] Lustre: Skipped 1 previous similar message [ 6119.654219] Lustre: server umount lustre-OST0000 complete [ 6129.228606] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88b382b56800.ost_server_uuid 50 [ 6130.847246] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6146.248965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.270644] Lustre: Skipped 1 previous similar message [ 6146.272910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6148.713406] Lustre: server umount lustre-MDT0000 complete [ 6151.660621] LustreError: 189455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6151.697098] LustreError: 189455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6153.845060] LustreError: 189437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972545 with bad export cookie 236051284974188807 [ 6153.870534] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6154.410840] Lustre: server umount lustre-MDT0001 complete [ 6162.400311] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:55:54 (1783972554) [ 6168.380578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.329674] LustreError: 193371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.448537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6174.650370] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6174.690191] LustreError: 193372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.202952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.492426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6189.298351] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6197.621657] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6203.252991] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6210.551602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6210.840111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6214.974517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6219.150660] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6228.839440] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6237.110584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.332537] Lustre: lustre-OST0001: new disk, initializing [ 6237.336636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6237.346308] Lustre: 196217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6237.452370] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6244.192929] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6245.575604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6245.588725] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6245.703387] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6254.948746] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6260.193710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6260.212080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.241593] Lustre: Skipped 1 previous similar message [ 6260.255225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6260.259809] Lustre: Skipped 1 previous similar message [ 6264.598773] Lustre: server umount lustre-OST0000 complete [ 6265.972896] LustreError: 195196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6266.001190] LustreError: 195196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6272.416211] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6272.736796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6278.421965] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6278.782400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6284.412725] LustreError: 195197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6284.452795] LustreError: 195197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6297.063394] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6297.067068] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.074364] LustreError: Skipped 1 previous similar message [ 6297.075418] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6297.075422] Lustre: Skipped 3 previous similar messages [ 6297.129499] Lustre: Skipped 2 previous similar messages [ 6299.602753] Lustre: server umount lustre-OST0001 complete [ 6307.312959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6307.323382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.358373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6307.367652] Lustre: Skipped 1 previous similar message [ 6309.723981] Lustre: server umount lustre-MDT0000 complete [ 6313.670769] LustreError: 193352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972705 with bad export cookie 236051284974189871 [ 6313.674920] LustreError: 193352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6313.679792] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.938737] Lustre: server umount lustre-MDT0001 complete [ 6338.860190] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:58:50 (1783972730) [ 6347.058915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.451680] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6347.508819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6347.764758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6353.708985] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6362.482179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.698971] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6362.782792] Lustre: 199655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6368.400542] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6377.022491] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6384.159632] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6391.433788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.629085] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6398.134799] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6400.612437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6405.466539] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6412.839402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6412.908167] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6413.112601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6413.118478] Lustre: Skipped 2 previous similar messages [ 6419.976782] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6420.586057] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6427.692280] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6431.201253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6431.215714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6431.221673] LustreError: Skipped 1 previous similar message [ 6431.225943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6431.253552] Lustre: Skipped 2 previous similar messages [ 6436.529992] Lustre: server umount lustre-OST0000 complete [ 6438.518862] LustreError: 201507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.539403] LustreError: 201507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6456.288903] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6456.302928] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.053592] Lustre: server umount lustre-OST0001 complete [ 6465.510602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6465.531486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6465.536938] Lustre: Skipped 6 previous similar messages [ 6470.389915] Lustre: server umount lustre-MDT0000 complete [ 6474.230238] LustreError: 199648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783972866 with bad export cookie 236051284974191285 [ 6474.245670] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6474.246487] LustreError: 199648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6474.595037] Lustre: server umount lustre-MDT0001 complete [ 6499.705417] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:01:31 (1783972891) [ 6517.371603] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 6529.132920] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6573.845497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6574.018716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6574.046614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6574.124438] Lustre: lustre-MDT0000: new disk, initializing [ 6574.234970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6574.270651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6579.923746] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6591.643567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.860971] Lustre: 207766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6591.968246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6591.971861] Lustre: Skipped 1 previous similar message [ 6592.094545] Lustre: lustre-MDT0001: new disk, initializing [ 6592.233409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6592.245342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6597.102775] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6606.900459] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6612.882811] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6619.981023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6620.281622] Lustre: lustre-OST0000: new disk, initializing [ 6620.286388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6620.296768] Lustre: 209733:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6626.596629] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6629.456304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6629.461821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6629.592826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6635.824664] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6640.027594] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6640.228591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6644.295602] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6644.565586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6650.344059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6650.351555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.356724] LustreError: Skipped 1 previous similar message [ 6650.357486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6650.357493] Lustre: Skipped 2 previous similar messages [ 6650.412208] Lustre: Skipped 4 previous similar messages [ 6652.747643] Lustre: server umount lustre-OST0000 complete [ 6670.303134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6670.539788] Lustre: server umount lustre-MDT0000 complete [ 6670.818475] LustreError: 207776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.830222] LustreError: 207776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6674.071882] LustreError: 209732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973065 with bad export cookie 236051284974193070 [ 6674.080584] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6674.093025] LustreError: 209732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6674.376561] Lustre: server umount lustre-MDT0001 complete [ 6683.482832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.064456] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6699.090552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.915944] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6713.805384] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6720.126717] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6726.699600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6733.023320] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6740.756078] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6745.837691] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6746.000396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6750.963306] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6751.257774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6753.763959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6753.776594] LustreError: Skipped 1 previous similar message [ 6753.780302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6753.795073] Lustre: Skipped 2 previous similar messages [ 6759.784327] Lustre: server umount lustre-OST0000 complete [ 6767.247432] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6772.111948] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6801.922032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.328384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6804.343644] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6808.765643] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6815.835092] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6819.662372] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6819.695334] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6819.717688] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6824.939120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6824.947175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6824.951465] LustreError: Skipped 2 previous similar messages [ 6824.962597] Lustre: Skipped 10 previous similar messages [ 6830.665214] Lustre: server umount lustre-OST0000 complete [ 6841.832908] Lustre: server umount lustre-MDT0000 complete [ 6845.413166] LustreError: 211953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.442116] LustreError: 211953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6846.553927] LustreError: 212688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973238 with bad export cookie 236051284974193952 [ 6846.558340] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6846.575435] LustreError: 212688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6846.945814] Lustre: server umount lustre-MDT0001 complete [ 6858.489820] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:07:30 (1783973250) [ 6865.438956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.888218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.893596] Lustre: Skipped 6 previous similar messages [ 6871.132391] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6880.538062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6885.584625] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6895.559425] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6901.669470] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6908.841446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6915.903382] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6923.407600] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6929.105191] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6929.392614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6934.568982] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6934.806765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6942.151756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.496768] Lustre: lustre-OST0001: new disk, initializing [ 6942.505724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6942.520912] Lustre: 221292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6949.972034] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 6950.023425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6950.036369] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6950.190228] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6958.376902] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6962.944708] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6963.157941] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6967.701539] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6967.930345] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6970.850465] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.851662] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.869395] LustreError: Skipped 1 previous similar message [ 6970.933741] Lustre: Skipped 6 previous similar messages [ 6984.159243] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6984.311284] Lustre: server umount lustre-OST0001 complete [ 6991.880663] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6992.109825] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6997.235632] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6997.455988] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7026.494285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7031.791080] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7031.808336] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7031.808825] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7031.824888] Lustre: Skipped 1 previous similar message [ 7032.739618] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7039.339954] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7040.683957] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7048.764904] Lustre: server umount lustre-OST0001 complete [ 7072.003763] Lustre: server umount lustre-OST0000 complete [ 7081.748914] Lustre: server umount lustre-MDT0000 complete [ 7085.282886] LustreError: 218655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973477 with bad export cookie 236051284974194911 [ 7085.296755] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7092.046320] Lustre: server umount lustre-MDT0001 complete [ 7120.202743] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:11:52 (1783973512) [ 7128.094371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.420751] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7128.439619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7134.701785] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7142.839574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.937225] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7142.970648] Lustre: 226536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7147.647947] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7155.399219] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7161.086461] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7167.862191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.194331] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7174.830224] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7183.823965] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7193.785235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7193.920278] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7201.413405] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7208.659807] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7210.305906] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7215.781799] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7216.008171] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7220.674741] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7220.842355] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7227.066619] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88b387e41800.ost_server_uuid 50 [ 7229.043648] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88b387e41800.ost_server_uuid in IDLE state after 0 sec [ 7230.115314] Lustre: Permanently deactivating lustre-OST0001 [ 7230.133930] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7233.529490] Lustre: Permanently reactivating lustre-OST0001 [ 7238.626478] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7238.627440] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.641876] LustreError: Skipped 3 previous similar messages [ 7238.644319] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7238.644326] Lustre: Skipped 20 previous similar messages [ 7238.694934] Lustre: Skipped 7 previous similar messages [ 7242.279808] Lustre: server umount lustre-OST0001 complete [ 7243.748283] LustreError: 228384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.781323] LustreError: 228384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 7252.624242] Lustre: server umount lustre-OST0000 complete [ 7263.052989] Lustre: server umount lustre-MDT0000 complete [ 7266.535168] LustreError: 226529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973658 with bad export cookie 236051284974196206 [ 7266.543647] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7266.547804] LustreError: 226529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7266.941271] Lustre: server umount lustre-MDT0001 complete [ 7290.224731] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:14:42 (1783973682) [ 7297.533760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.868849] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7297.925182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7297.939823] Lustre: Skipped 5 previous similar messages [ 7303.969578] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7312.857605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.957865] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7312.978555] Lustre: 233231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7318.542500] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7326.679409] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7332.985608] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7340.272235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7340.457422] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7340.475368] Lustre: Permanently deactivating lustre-OST0000 [ 7340.483697] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7346.766051] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7347.941584] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7356.381661] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7364.263502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7364.389518] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7370.979828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7371.270853] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7379.164268] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7381.361844] Lustre: Permanently reactivating lustre-OST0000 [ 7381.376487] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7381.387274] Lustre: Skipped 2 previous similar messages [ 7387.723557] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7387.726071] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 000000004154e92d, cur 1783973779 deadline 1783973768 last 1783973738 [ 7387.738347] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7387.741272] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7387.804900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7387.833366] LustreError: 235399:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -107 [ 7388.903852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7403.373487] Lustre: server umount lustre-OST0001 complete [ 7414.583523] Lustre: server umount lustre-OST0000 complete [ 7433.183134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7433.570198] Lustre: server umount lustre-MDT0000 complete [ 7438.817381] LustreError: 236826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783973830 with bad export cookie 236051284974197851 [ 7438.818869] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7438.835721] LustreError: 236826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7439.570909] Lustre: server umount lustre-MDT0001 complete [ 7451.733174] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 7455.166941] Key type lgssc unregistered [ 7455.496208] LNet: 238116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7455.500840] LNetError: 238116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7455.515839] LNet: Removed LNI 192.168.202.114@tcp [ 7456.280964] Key type .llcrypt unregistered [ 7456.283205] Key type ._llcrypt unregistered [ 7484.361373] Key type ._llcrypt registered [ 7484.366322] Key type .llcrypt registered [ 7484.544665] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7486.667199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7486.986887] alg: No test for adler32 (adler32-zlib) [ 7488.295230] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 7488.881569] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 7490.743196] Key type lgssc registered [ 7492.715193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7505.227209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7505.253498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.941734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.569739] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7516.140574] LustreError: 239584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7516.154012] LustreError: 239584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7521.264656] LustreError: 239585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7522.460719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.663779] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7522.722447] Lustre: 239572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7522.742146] Lustre: Permanently deactivating lustre-MDT0001 [ 7522.761419] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7523.099274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7527.877368] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7531.603677] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7535.907414] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7542.516627] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7549.702353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.131793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7551.152256] LustreError: 241424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.181304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7556.590019] LustreError: 241497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7556.972475] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7561.708026] LustreError: 241426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.631505] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7567.866514] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7571.834777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7572.002225] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7577.088540] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7577.129092] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7579.491745] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7589.137368] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7592.474302] Lustre: Permanently reactivating lustre-MDT0001 [ 7592.493329] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7592.495586] Lustre: Skipped 2 previous similar messages [ 7598.202844] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7598.212094] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7598.220761] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7598.229802] LustreError: 243155:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7598.230855] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7598.238588] LustreError: 243155:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7598.248649] LustreError: 243155:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.114@tcp: cfg command failed: rc = -108 [ 7598.254308] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7598.254308] [ 7598.259479] LustreError: 239576:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7599.260618] Lustre: lustre-MDT0001: Client 9b65ec21-dbcd-4cda-b202-79ba6b715d45 (at 192.168.202.14@tcp) reconnecting [ 7604.628225] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7606.481478] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7613.263677] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7613.475412] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7618.203297] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7618.548518] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7620.960430] Lustre: Permanently deactivating lustre-MDT0001 [ 7620.993529] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7621.001582] Lustre: Skipped 2 previous similar messages [ 7629.422563] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7641.058795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7641.068699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7641.096106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7644.646734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.655661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7644.694807] Lustre: Skipped 2 previous similar messages [ 7644.701288] Lustre: Skipped 2 previous similar messages [ 7646.284384] Lustre: server umount lustre-MDT0000 complete [ 7649.770436] LustreError: 239585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7649.791609] LustreError: 239585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7651.850310] LustreError: 242439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974043 with bad export cookie 13485743566859576354 [ 7651.854346] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7651.858771] LustreError: 242439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7652.353913] Lustre: server umount lustre-MDT0001 complete [ 7659.939251] Lustre: server umount lustre-OST0000 complete [ 7670.829094] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:21:02 (1783974062) [ 7680.681085] Lustre: server umount lustre-OST0001 complete [ 7687.374917] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 7698.384936] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7740.984745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.315857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7741.340685] Lustre: Skipped 2 previous similar messages [ 7741.398338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7741.571695] Lustre: lustre-MDT0000: new disk, initializing [ 7741.732846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7741.774330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7747.527398] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7758.948260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7759.133857] Lustre: 247898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7759.215800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7759.220927] Lustre: Skipped 1 previous similar message [ 7759.322211] Lustre: lustre-MDT0001: new disk, initializing [ 7759.483531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7759.537293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7759.554215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7764.741360] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7774.597910] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7780.801277] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7789.076412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7789.464401] Lustre: lustre-OST0000: new disk, initializing [ 7789.469920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7789.487422] Lustre: 249865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7789.647511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7797.199430] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7799.374933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7799.396048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7799.554346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7807.604659] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7809.723796] LustreError: 247903:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7819.674490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.801043] LustreError: 247903:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7819.809672] Lustre: lustre-OST0001: new disk, initializing [ 7819.815668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7819.823847] LustreError: 247904:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7819.826512] Lustre: 250980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7825.545297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7825.565524] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7825.731853] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7827.482406] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 7829.864060] LustreError: 247904:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7839.018236] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7841.496805] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7846.396211] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7846.402942] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.412503] LustreError: Skipped 1 previous similar message [ 7846.459200] Lustre: Skipped 1 previous similar message [ 7846.471832] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7846.490661] Lustre: Skipped 1 previous similar message [ 7850.557497] Lustre: server umount lustre-OST0001 complete [ 7851.495428] LustreError: 249851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7851.512974] LustreError: 249851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7871.456449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7871.459717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.463807] LustreError: Skipped 1 previous similar message [ 7871.478813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7871.489832] Lustre: Skipped 1 previous similar message [ 7877.097982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7877.104533] Lustre: Skipped 1 previous similar message [ 7884.257163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7884.579226] Lustre: server umount lustre-OST0000 complete [ 7892.449629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7892.451715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7892.455673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7892.481348] Lustre: Skipped 2 previous similar messages [ 7892.500096] Lustre: Skipped 3 previous similar messages [ 7895.446404] Lustre: server umount lustre-MDT0000 complete [ 7897.568439] LustreError: 247904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.607195] LustreError: 247904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 7899.613976] LustreError: 248706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974291 with bad export cookie 13485743566859579168 [ 7899.616236] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7899.618770] LustreError: 248706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7905.870908] Lustre: server umount lustre-MDT0001 complete [ 7922.547441] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 7927.274047] Key type lgssc unregistered [ 7927.661190] LNet: 253017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7927.674052] LNetError: 253017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7927.693663] LNet: Removed LNI 192.168.202.114@tcp [ 7928.605861] Key type .llcrypt unregistered [ 7928.611274] Key type ._llcrypt unregistered [ 7972.967386] Key type ._llcrypt registered [ 7972.974415] Key type .llcrypt registered [ 7973.164685] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7974.468134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7974.644175] alg: No test for adler32 (adler32-zlib) [ 7975.995505] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 7976.424901] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 7978.319180] Key type lgssc registered [ 7980.153795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7989.699645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7989.733184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7991.151773] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7991.182074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7991.348516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7995.806797] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8005.443202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8005.514868] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8005.548387] Lustre: 255228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8005.811821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8011.864781] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8020.794953] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8026.524030] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8034.796592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8035.023341] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8035.314898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8041.108542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8041.403809] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8049.148589] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8107.488426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8107.497358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8107.520957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8108.005972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8108.015302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8108.028629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8108.032249] Lustre: Skipped 1 previous similar message [ 8110.143055] Lustre: server umount lustre-OST0000 complete [ 8115.091659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8128.362800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8129.020151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.145573] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8134.180335] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8136.767726] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8144.941651] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8148.521316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8148.561816] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8184.804303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8184.813910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.832138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8189.412727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8189.424133] Lustre: Skipped 2 previous similar messages [ 8191.251277] Lustre: server umount lustre-OST0000 complete [ 8199.651695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.655638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8199.696129] Lustre: Skipped 2 previous similar messages [ 8199.722647] Lustre: Skipped 1 previous similar message [ 8209.889509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.928547] Lustre: Skipped 3 previous similar messages [ 8210.399923] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8210.766336] Lustre: server umount lustre-MDT0000 complete [ 8215.986550] LustreError: 257089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974607 with bad export cookie 9044470790733373779 [ 8215.992484] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8216.230927] Lustre: server umount lustre-MDT0001 complete [ 8226.552960] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 8229.771312] Key type lgssc unregistered [ 8230.045244] LNet: 260656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8230.062795] LNetError: 260656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8230.095343] LNet: Removed LNI 192.168.202.114@tcp [ 8231.216172] Key type .llcrypt unregistered [ 8231.218822] Key type ._llcrypt unregistered [ 8256.738271] Key type ._llcrypt registered [ 8256.740722] Key type .llcrypt registered [ 8256.829376] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8257.763060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8257.916886] alg: No test for adler32 (adler32-zlib) [ 8259.016780] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 8259.330320] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 8261.111815] Key type lgssc registered [ 8262.395057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8270.556944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8270.585693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.274963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8278.294051] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8282.595763] LustreError: 262075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8282.602141] LustreError: 262075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8287.721750] LustreError: 262076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8289.561395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.088075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8294.871429] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8305.375923] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8315.121453] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8321.173664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8321.511290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.405578] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8336.783709] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8343.023062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8365.539283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8365.549759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.591743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8368.618322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8368.625989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8371.718802] Lustre: server umount lustre-OST0000 complete [ 8377.314042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8377.324362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8377.354847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8377.372931] Lustre: Skipped 1 previous similar message [ 8381.661725] Lustre: server umount lustre-MDT0000 complete [ 8383.972368] LustreError: 262076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8383.982796] LustreError: 262076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8385.389479] LustreError: 262056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974777 with bad export cookie 18422861735172601425 [ 8385.402628] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8385.678684] Lustre: server umount lustre-MDT0001 complete [ 8396.932301] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 8400.069248] Key type lgssc unregistered [ 8400.387936] LNet: 266383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8400.400223] LNetError: 266383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8400.433379] LNet: Removed LNI 192.168.202.114@tcp [ 8401.448105] Key type .llcrypt unregistered [ 8401.454794] Key type ._llcrypt unregistered [ 8423.796785] Key type ._llcrypt registered [ 8423.798358] Key type .llcrypt registered [ 8423.972401] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8425.912673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8426.179392] alg: No test for adler32 (adler32-zlib) [ 8427.429866] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 8427.778202] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 8429.423524] Key type lgssc registered [ 8430.995421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8439.050175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8439.089151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.654577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8444.716301] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8451.041950] LustreError: 267610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8451.073369] LustreError: 267610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8454.071134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.659418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8459.159538] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8465.989809] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8471.008566] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8478.636520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8478.981649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.251654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8487.152471] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8496.830269] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8510.673387] Lustre: server umount lustre-OST0000 complete [ 8516.064834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.078300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.093888] Lustre: Skipped 1 previous similar message [ 8521.192781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8526.239570] Lustre: 266922:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783974902/real 1783974902] req@00000000f3e43343 x1870633179836928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783974918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8526.261433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.265985] Lustre: Skipped 1 previous similar message [ 8526.269288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8526.272268] Lustre: Skipped 1 previous similar message [ 8527.327605] Lustre: 266921:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783974902/real 1783974902] req@0000000046fe32cf x1870633179837312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783974918 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8529.378234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8529.792571] Lustre: server umount lustre-MDT0000 complete [ 8531.423384] Lustre: 266922:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783974907/real 1783974907] req@000000000ba1c262 x1870633179838720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783974923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8531.432134] LustreError: 268877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.490235] LustreError: 268877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8534.437332] LustreError: 267592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783974926 with bad export cookie 4093805929054233159 [ 8534.452216] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8534.839534] Lustre: server umount lustre-MDT0001 complete [ 8548.991120] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 8553.784803] Key type lgssc unregistered [ 8554.284650] LNet: 271288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8554.299987] LNetError: 271288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8554.333430] LNet: Removed LNI 192.168.202.114@tcp [ 8555.703655] Key type .llcrypt unregistered [ 8555.706055] Key type ._llcrypt unregistered [ 8580.349327] Key type ._llcrypt registered [ 8580.351199] Key type .llcrypt registered [ 8580.445524] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8582.391465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8582.462336] alg: No test for adler32 (adler32-zlib) [ 8583.945836] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 8584.270039] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 8586.023438] Key type lgssc registered [ 8587.808859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8598.508936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8598.549588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.414911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.155594] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8610.784571] LustreError: 272687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.818774] LustreError: 272687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8615.692037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8615.905085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8616.136640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8622.538176] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8631.522214] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8638.898377] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8647.229638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8647.759813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8654.821389] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8663.712186] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8671.365434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8702.432184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8702.442376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8702.445055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8702.459258] Lustre: Skipped 1 previous similar message [ 8702.468778] Lustre: Skipped 1 previous similar message [ 8707.380233] Lustre: server umount lustre-OST0000 complete [ 8712.672176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8712.676680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8712.699657] Lustre: Skipped 1 previous similar message [ 8712.708376] Lustre: Skipped 1 previous similar message [ 8717.856575] LustreError: 272684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.877889] LustreError: 272684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8717.960973] Lustre: server umount lustre-MDT0000 complete [ 8722.915266] LustreError: 273415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8722.973882] LustreError: 273415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8723.972172] LustreError: 272668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975115 with bad export cookie 2401154871626773413 [ 8723.993357] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8724.672866] Lustre: server umount lustre-MDT0001 complete [ 8742.699156] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 8746.113572] Key type lgssc unregistered [ 8746.411879] LNet: 277031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8746.420446] LNetError: 277031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8746.445262] LNet: Removed LNI 192.168.202.114@tcp [ 8747.434092] Key type .llcrypt unregistered [ 8747.437339] Key type ._llcrypt unregistered [ 8767.585778] Key type ._llcrypt registered [ 8767.592439] Key type .llcrypt registered [ 8767.812371] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8769.393922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8769.421804] alg: No test for adler32 (adler32-zlib) [ 8770.626892] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 8771.025924] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 8772.809108] Key type lgssc registered [ 8774.429150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8784.255385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8784.274655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.979493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.100744] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8795.132933] LustreError: 278272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.159027] LustreError: 278272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8800.236165] LustreError: 278274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8800.602902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.206764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8806.698629] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8816.735912] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8823.255362] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8831.207125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8831.763901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8839.671102] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 8849.184789] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8855.414944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8876.005916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8876.011164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8876.022733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8876.058503] Lustre: Skipped 1 previous similar message [ 8878.319832] Lustre: server umount lustre-OST0000 complete [ 8883.168520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8883.171831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8883.214936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8883.223167] Lustre: Skipped 1 previous similar message [ 8886.245505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8886.247450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8889.221143] Lustre: server umount lustre-MDT0000 complete [ 8891.380953] LustreError: 278256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8891.415296] LustreError: 278256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8893.636328] LustreError: 278240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975285 with bad export cookie 3195455612124056645 [ 8893.658229] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8894.223341] Lustre: server umount lustre-MDT0001 complete [ 8905.846078] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 8909.248170] Key type lgssc unregistered [ 8909.616043] LNet: 281994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8909.629219] LNetError: 281994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8909.651691] LNet: Removed LNI 192.168.202.114@tcp [ 8910.814581] Key type .llcrypt unregistered [ 8910.817125] Key type ._llcrypt unregistered [ 8927.499689] Key type ._llcrypt registered [ 8927.502511] Key type .llcrypt registered [ 8927.694339] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8948.191376] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 8964.646887] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 8966.501634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8966.565976] alg: No test for adler32 (adler32-zlib) [ 8967.900226] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 8968.173448] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 8969.847214] Key type lgssc registered [ 8970.992284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9010.167940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9010.190156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9011.512888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9011.541936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9011.633897] Lustre: lustre-MDT0000: new disk, initializing [ 9011.735728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.760753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9016.578660] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9029.746030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.938806] Lustre: 285545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9030.068809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9030.078224] Lustre: Skipped 1 previous similar message [ 9030.250070] Lustre: lustre-MDT0001: new disk, initializing [ 9030.463506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9030.569373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9030.590660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9038.373346] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9048.560847] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9054.698712] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9061.306049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9061.917855] Lustre: lustre-OST0000: new disk, initializing [ 9061.924435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9061.931270] Lustre: 287514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9062.101127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.732727] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9071.710311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9071.743164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9071.896455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9080.475326] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9086.452437] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9086.753425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9092.647414] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9092.974590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9097.704183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9097.705569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9097.712964] LustreError: Skipped 1 previous similar message [ 9097.714515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9097.750687] Lustre: Skipped 1 previous similar message [ 9101.584845] Lustre: server umount lustre-OST0000 complete [ 9107.425159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9107.430809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.443595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9107.446274] Lustre: Skipped 1 previous similar message [ 9112.626758] Lustre: server umount lustre-MDT0000 complete [ 9116.739929] LustreError: 285537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975508 with bad export cookie 12496519637381682094 [ 9116.749710] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9116.761150] LustreError: 285537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9117.165578] Lustre: server umount lustre-MDT0001 complete [ 9128.185590] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:45:19 (1783975519) [ 9135.757933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.213014] LustreError: 289929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9136.233763] LustreError: 289929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9136.285406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9141.339195] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9141.734148] LustreError: 289930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9146.859482] LustreError: 289929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9150.270714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.669393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9155.880428] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9164.889410] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9171.023333] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9178.607694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9179.050352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9186.967296] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9196.412318] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9207.778083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9207.790434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9207.801512] Lustre: Skipped 1 previous similar message [ 9207.811134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9207.821309] Lustre: Skipped 1 previous similar message [ 9210.848786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9210.850578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9210.864519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.177456] Lustre: server umount lustre-OST0000 complete [ 9217.505150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9217.511309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9217.521132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9217.524086] Lustre: Skipped 1 previous similar message [ 9226.217755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9226.229807] Lustre: Skipped 3 previous similar messages [ 9231.327567] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9231.867580] Lustre: server umount lustre-MDT0000 complete [ 9235.427515] LustreError: 289924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.447962] LustreError: 289924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9237.067136] LustreError: 292514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975628 with bad export cookie 12496519637381682976 [ 9237.076312] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9237.086614] LustreError: 292514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9237.866380] Lustre: server umount lustre-MDT0001 complete [ 9251.154985] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 9254.596746] Key type lgssc unregistered [ 9254.758989] LNet: 293471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9254.767437] LNetError: 293471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9254.792732] LNet: Removed LNI 192.168.202.114@tcp [ 9255.892785] Key type .llcrypt unregistered [ 9255.896547] Key type ._llcrypt unregistered [ 9286.290926] Key type ._llcrypt registered [ 9286.295363] Key type .llcrypt registered [ 9286.378585] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 9306.915632] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9308.799620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9308.833181] alg: No test for adler32 (adler32-zlib) [ 9310.109153] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 9310.450290] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 9312.271622] Key type lgssc registered [ 9313.486740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9365.976624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9366.009129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9367.305893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9367.333914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9367.459745] Lustre: lustre-MDT0000: new disk, initializing [ 9367.556860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9367.598395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9372.853784] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9384.835289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9385.046304] Lustre: 297299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9385.101204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9385.105573] Lustre: Skipped 1 previous similar message [ 9385.170380] Lustre: lustre-MDT0001: new disk, initializing [ 9385.258949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9385.300259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9385.327352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9390.492723] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9399.779682] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9406.403945] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9413.704121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9413.993428] Lustre: lustre-OST2710: new disk, initializing [ 9413.997527] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9414.003924] Lustre: 299268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9414.115965] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9421.051796] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9423.425558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9423.451690] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9423.564428] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9430.808980] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9438.054323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9438.283940] Lustre: lustre-OST03e8: new disk, initializing [ 9438.294902] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9438.300311] Lustre: 300331:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9438.395812] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9445.294704] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9448.090386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9448.101989] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9448.193379] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9455.154573] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9464.346729] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9464.660834] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9470.830729] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9471.052894] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9476.943838] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9477.312661] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9482.556641] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9482.761178] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9487.845138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9487.862913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9487.889814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9489.379581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9489.382862] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.391844] Lustre: Skipped 2 previous similar messages [ 9492.871862] Lustre: server umount lustre-MDT0000 complete [ 9496.812371] LustreError: 300330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783975888 with bad export cookie 17976066355978539655 [ 9496.823717] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9496.825207] LustreError: 300330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9497.330483] Lustre: server umount lustre-MDT0001 complete [ 9512.074154] Lustre: server umount lustre-OST2710 complete [ 9525.920643] Lustre: server umount lustre-OST03e8 complete [ 9546.473682] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 9559.569889] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9599.727029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9599.939431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9599.970678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9600.080691] Lustre: lustre-MDT0000: new disk, initializing [ 9600.193283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9600.216486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9605.815824] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9619.060706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.280990] Lustre: 305587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9619.380730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9619.386590] Lustre: Skipped 1 previous similar message [ 9619.505513] Lustre: lustre-MDT0001: new disk, initializing [ 9619.740616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9619.813439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9619.832060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9625.111648] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9634.949889] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9642.598509] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9650.428568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9650.665623] Lustre: lustre-OST0000: new disk, initializing [ 9650.668216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9650.673247] Lustre: 307556:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9650.751571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9658.448131] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9659.468694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9659.480455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9659.605954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9669.837806] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9675.817209] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9676.254240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9681.964894] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9682.321933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9685.472797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9685.490931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9685.521585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9685.549450] Lustre: Skipped 4 previous similar messages [ 9688.545055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9688.553160] Lustre: Skipped 1 previous similar message [ 9690.936470] Lustre: server umount lustre-OST0000 complete [ 9696.741424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9696.749076] LustreError: Skipped 1 previous similar message [ 9696.756829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.774431] Lustre: Skipped 1 previous similar message [ 9696.777393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9702.889885] Lustre: server umount lustre-MDT0000 complete [ 9703.904901] LustreError: 305598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.930800] LustreError: 305598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9708.277736] LustreError: 305578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976100 with bad export cookie 17976066355978541342 [ 9708.278874] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9708.296856] LustreError: 305578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9708.720423] Lustre: server umount lustre-MDT0001 complete [ 9719.854611] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:55:11 (1783976111) [ 9721.840526] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9724.000351] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:55:15 (1783976115) [ 9728.263786] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9762.727841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9763.409939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9763.439538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9763.679837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9769.269917] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9779.698559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.769943] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9779.792772] Lustre: 311417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9784.562337] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9807.798500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9808.164103] Lustre: Denying initial registration attempt from nid 192.168.202.114@tcp, specified as failover [ 9808.172456] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9808.180608] LustreError: 312871:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9808.185946] LustreError: 312871:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9808.190776] LustreError: 312871:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9808.274763] Lustre: server umount lustre-OST0000 complete [ 9808.277707] LustreError: 312871:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9820.147351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.185723] Lustre: Skipped 1 previous similar message [ 9820.195437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9820.201941] Lustre: Skipped 3 previous similar messages [ 9822.715964] Lustre: server umount lustre-MDT0000 complete [ 9825.248178] LustreError: 311429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9825.276499] LustreError: 311429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9827.023820] LustreError: 311409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976218 with bad export cookie 17976066355978542224 [ 9827.041134] LustreError: 311409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9827.042208] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9827.355540] Lustre: server umount lustre-MDT0001 complete [ 9839.808547] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 9843.093181] Key type lgssc unregistered [ 9843.443728] LNet: 313991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9843.453299] LNetError: 313991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9843.475370] LNet: Removed LNI 192.168.202.114@tcp [ 9844.509319] Key type .llcrypt unregistered [ 9844.512099] Key type ._llcrypt unregistered [ 9857.649544] Key type ._llcrypt registered [ 9857.651330] Key type .llcrypt registered [ 9857.793869] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9859.403618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9859.438272] alg: No test for adler32 (adler32-zlib) [ 9860.739994] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [ 9861.019427] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 9862.767168] Key type lgssc registered [ 9864.303431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9916.195481] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 9927.598828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9927.626523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9929.030722] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9929.078662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9929.257208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9934.799594] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9944.358741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.479857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9944.527914] Lustre: 316759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9944.848830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9950.840212] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [ 9959.702616] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9967.045589] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9987.828760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9988.144244] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9988.326412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9995.511345] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10004.792458] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10008.036386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10008.049091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10008.070398] Lustre: Skipped 1 previous similar message [10008.096001] Lustre: Skipped 1 previous similar message [10010.467698] Lustre: server umount lustre-OST0000 complete [10016.737071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10016.742081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.771650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10018.277796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.279493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10018.314672] Lustre: Skipped 1 previous similar message [10021.458431] Lustre: server umount lustre-MDT0000 complete [10025.764683] LustreError: 316751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976417 with bad export cookie 17416585049049794000 [10025.777755] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10025.781934] LustreError: 316751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10026.372939] Lustre: server umount lustre-MDT0001 complete [10039.464367] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10042.541346] Key type lgssc unregistered [10042.921280] LNet: 320334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10042.925266] LNetError: 320334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10042.936215] LNet: Removed LNI 192.168.202.114@tcp [10043.946248] Key type .llcrypt unregistered [10043.948376] Key type ._llcrypt unregistered [10074.175690] Key type ._llcrypt registered [10074.184182] Key type .llcrypt registered [10074.381394] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10076.023825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10076.092553] alg: No test for adler32 (adler32-zlib) [10077.165866] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [10077.643682] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [10079.391228] Key type lgssc registered [10081.167960] Lustre: Echo OBD driver; http://www.lustre.org/ [10091.300201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10091.339679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.062174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10099.850266] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10102.243677] LustreError: 321761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.269662] LustreError: 321761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10107.361115] LustreError: 321762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.826341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.154658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10115.012066] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10124.028732] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10131.273701] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10142.309525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10142.661370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10148.938256] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10157.100451] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10175.968409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10175.977339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10175.987209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10178.189383] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.14@tcp (stopping) [10178.534523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10181.827786] Lustre: server umount lustre-MDT0000 complete [10183.280635] LustreError: 323606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10186.513561] LustreError: 321742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976578 with bad export cookie 13450199066843147721 [10186.538269] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10188.398438] LustreError: 321758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.435877] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [10188.440631] LustreError: 321758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10188.463020] Lustre: Skipped 3 previous similar messages [10188.771030] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10188.790086] Lustre: Skipped 1 previous similar message [10191.777131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10191.794597] Lustre: Skipped 1 previous similar message [10193.457983] Lustre: server umount lustre-MDT0001 complete [10201.063466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10212.423899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10232.268715] LustreError: 323596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.300352] LustreError: 323596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10232.475192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10232.583347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10237.574492] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10242.017433] LustreError: 325382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.030535] LustreError: 325382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10246.117627] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10246.951124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.306962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10251.823450] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10252.800187] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10260.063667] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10261.552438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10267.737092] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10269.271434] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10277.856781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10277.865519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.877783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10280.977328] Lustre: server umount lustre-OST0000 complete [10288.108684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10288.119484] LustreError: Skipped 1 previous similar message [10288.124387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10288.144409] Lustre: Skipped 1 previous similar message [10288.156759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10288.161734] Lustre: Skipped 2 previous similar messages [10290.742856] Lustre: server umount lustre-MDT0000 complete [10293.729059] LustreError: 325382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10293.752301] LustreError: 325382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10294.672668] LustreError: 326124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976686 with bad export cookie 13450199066843155561 [10294.697126] LustreError: 326124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10294.697384] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10295.298186] Lustre: server umount lustre-MDT0001 complete [10307.780184] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10311.336046] Key type lgssc unregistered [10311.709247] LNet: 328057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10311.718399] LNetError: 328057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10311.739598] LNet: Removed LNI 192.168.202.114@tcp [10313.038000] Key type .llcrypt unregistered [10313.044474] Key type ._llcrypt unregistered [10342.718981] Key type ._llcrypt registered [10342.721675] Key type .llcrypt registered [10342.882664] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10343.979217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10344.048231] alg: No test for adler32 (adler32-zlib) [10345.412765] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [10345.807726] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [10347.575196] Key type lgssc registered [10349.131900] Lustre: Echo OBD driver; http://www.lustre.org/ [10357.143662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10357.168089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.918275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10363.799491] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10368.483506] LustreError: 329476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10368.520892] LustreError: 329476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10372.447805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10372.894634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10377.243386] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10383.631465] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10389.657062] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10393.569157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10393.575807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.580029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10393.586010] Lustre: Skipped 1 previous similar message [10393.592437] Lustre: Skipped 1 previous similar message [10397.729093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10397.731811] Lustre: Skipped 1 previous similar message [10399.824906] Lustre: server umount lustre-MDT0000 complete [10402.791017] LustreError: 329477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.805134] LustreError: 329477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10403.762719] LustreError: 329456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976795 with bad export cookie 12368476417390150803 [10403.790532] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10404.270823] Lustre: server umount lustre-MDT0001 complete [10419.224036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10419.514096] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10419.564227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10419.830613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10425.772837] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10434.749487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10434.915522] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10434.996313] Lustre: 332063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10435.295261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10440.700289] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10451.036527] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10457.085190] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10467.258991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10467.644272] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10467.652283] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10467.661557] LustreError: 333948:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10467.667248] LustreError: 333948:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10467.671924] LustreError: 333948:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10467.765622] Lustre: server umount lustre-OST0000 complete [10467.770334] LustreError: 333948:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10478.267503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10478.593864] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10478.857570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10486.105745] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10487.963228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10493.504521] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10501.571936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10501.665630] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10501.750480] Lustre: lustre-OST0001: new disk, initializing [10501.755730] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10501.761997] Lustre: 335490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10501.897371] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10508.522124] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10508.924522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10508.941402] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10509.082270] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10517.480790] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10524.649288] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10524.652565] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.654886] LustreError: Skipped 1 previous similar message [10524.657931] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10524.683757] Lustre: Skipped 1 previous similar message [10529.768469] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10529.774193] Lustre: Skipped 2 previous similar messages [10534.903866] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10535.391842] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10535.603382] Lustre: server umount lustre-OST0001 complete [10539.500068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10539.505893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10540.009041] LustreError: 334414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10540.039854] LustreError: 334414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10545.124789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10545.125098] LustreError: 335873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10545.141078] Lustre: Skipped 4 previous similar messages [10545.187545] LustreError: 335873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10545.888812] Lustre: server umount lustre-OST0000 complete [10553.312499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10553.323318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.343269] Lustre: Skipped 1 previous similar message [10556.733555] Lustre: server umount lustre-MDT0000 complete [10560.482730] LustreError: 332074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10560.518373] LustreError: 332074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10560.681449] LustreError: 332054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783976952 with bad export cookie 12368476417390151356 [10560.687642] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10560.693042] LustreError: 332054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10561.012729] Lustre: server umount lustre-MDT0001 complete [10572.385832] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [10575.537801] Key type lgssc unregistered [10575.896031] LNet: 337451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10575.906673] LNetError: 337451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10575.929085] LNet: Removed LNI 192.168.202.114@tcp [10576.876643] Key type .llcrypt unregistered [10576.881471] Key type ._llcrypt unregistered [10654.024281] Key type ._llcrypt registered [10654.027492] Key type .llcrypt registered [10654.175772] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [10670.664341] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [10672.096885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10672.164617] alg: No test for adler32 (adler32-zlib) [10673.276252] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [10673.771913] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [10675.655374] Key type lgssc registered [10677.910750] Lustre: Echo OBD driver; http://www.lustre.org/ [10724.839000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10724.916480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10726.283646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10726.341162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10726.473191] Lustre: lustre-MDT0000: new disk, initializing [10726.561688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10726.581498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10732.099245] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10744.933569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10745.061454] Lustre: 342389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10745.113544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10745.117758] Lustre: Skipped 1 previous similar message [10745.228971] Lustre: lustre-MDT0001: new disk, initializing [10745.359452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10745.414604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10745.428776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10751.606174] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10762.976055] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10769.709713] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10777.499102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10777.944894] Lustre: lustre-OST0000: new disk, initializing [10777.949507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10777.965765] Lustre: 344358:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10778.110706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10783.268209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10783.281755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10783.493602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10784.979059] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10796.248090] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10802.437762] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10802.729759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10808.633696] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10808.981794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10814.436435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10814.438112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10814.454242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10814.484487] Lustre: Skipped 1 previous similar message [10817.743757] Lustre: server umount lustre-OST0000 complete [10822.112338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10822.117769] LustreError: Skipped 1 previous similar message [10822.120879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10822.130273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10822.134473] Lustre: Skipped 1 previous similar message [10824.682665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10824.684421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10824.714290] Lustre: Skipped 1 previous similar message [10828.594294] Lustre: server umount lustre-MDT0000 complete [10832.779101] LustreError: 343193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977224 with bad export cookie 2706576527227309211 [10832.792165] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10833.374785] Lustre: server umount lustre-MDT0001 complete [10842.627914] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:13:54 (1783977234) [10851.105342] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:14:02 (1783977242) [10860.344376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.131494] LustreError: 347117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.169818] LustreError: 347117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10861.251378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10866.034714] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10866.668038] LustreError: 347118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10871.777831] LustreError: 347117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.894267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10875.372221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10880.279403] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10889.034742] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10895.303951] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10902.788357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10903.112988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10910.362715] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10918.782285] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10927.614724] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10929.584398] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10931.498797] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10933.443975] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10935.679908] Lustre: Failing over lustre-MDT0000 [10936.135485] Lustre: server umount lustre-MDT0000 complete [10936.303367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10936.311212] LustreError: 347114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10936.332500] Lustre: Skipped 1 previous similar message [10936.358100] LustreError: 347114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10941.411245] LustreError: 347845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.442237] LustreError: 347845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10951.649084] LustreError: 347117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.674263] LustreError: 347117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10952.673291] Lustre: 340611:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783977328/real 1783977328] req@00000000e343b9ab x1870635534982656/t0(0) o400->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783977344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10952.737705] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10956.597826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10963.375210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10963.446434] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10963.579748] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10968.046625] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [10968.558106] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10968.604978] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10968.674830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10978.868253] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10980.592547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10984.480990] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10986.570048] Lustre: Failing over lustre-MDT0000 [10987.057159] Lustre: server umount lustre-MDT0000 complete [10989.027406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10989.030086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10989.034702] Lustre: Skipped 1 previous similar message [10989.035531] LustreError: 347114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10989.035538] LustreError: 347114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [11004.384507] Lustre: 340613:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783977380/real 1783977380] req@00000000fabf9d59 x1870635535004416/t0(0) o400->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783977396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11004.423208] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11007.929823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11014.633194] LustreError: 340609:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004e88349c x1870635535010048/t0(0) o250->MGC192.168.202.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11015.019167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11015.098887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11015.793597] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11019.992138] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11020.275953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11020.289027] Lustre: Skipped 2 previous similar messages [11020.320435] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11020.379074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11031.378301] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11033.056447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11040.752336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11040.755975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11040.759298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11040.771212] Lustre: Skipped 4 previous similar messages [11040.773697] Lustre: Skipped 1 previous similar message [11045.875354] LustreError: 347118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11045.894226] LustreError: 347118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [11046.040604] Lustre: server umount lustre-MDT0000 complete [11049.938712] LustreError: 347100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977441 with bad export cookie 2706576527227311920 [11049.950317] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11049.964531] LustreError: 347100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11050.306311] Lustre: server umount lustre-MDT0001 complete [11063.570053] Lustre: server umount lustre-OST0000 complete [11079.730858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.405111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11085.527445] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11095.452115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.488855] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11109.902901] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11115.966804] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11123.431209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11125.876240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11130.602453] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11139.113787] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11150.816947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11150.830085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11150.851112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11150.853315] Lustre: Skipped 1 previous similar message [11150.856745] Lustre: Skipped 1 previous similar message [11155.196396] Lustre: server umount lustre-OST0000 complete [11161.062251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11161.072786] Lustre: Skipped 2 previous similar messages [11173.343819] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11173.700719] Lustre: server umount lustre-MDT0000 complete [11175.394931] LustreError: 353459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.413888] LustreError: 353459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11178.563397] LustreError: 353439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977570 with bad export cookie 2706576527227312662 [11178.564740] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11178.579215] LustreError: 353439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11179.048447] Lustre: server umount lustre-MDT0001 complete [11191.223965] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [11194.349979] Key type lgssc unregistered [11194.649838] LNet: 356994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11194.665906] LNetError: 356994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11194.675846] LNet: Removed LNI 192.168.202.114@tcp [11195.690972] Key type .llcrypt unregistered [11195.692836] Key type ._llcrypt unregistered [11222.228369] Key type ._llcrypt registered [11222.234214] Key type .llcrypt registered [11222.343753] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [11223.583865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11223.595840] alg: No test for adler32 (adler32-zlib) [11224.623495] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [11224.822489] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [11226.751790] Key type lgssc registered [11228.413440] Lustre: Echo OBD driver; http://www.lustre.org/ [11238.229919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11238.258145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11240.183484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11244.828508] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11249.638277] LustreError: 358511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11249.660717] LustreError: 358511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11254.765305] LustreError: 358512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11256.053250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11256.480044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11261.513678] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11269.657345] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11275.977270] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11282.664069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11283.027282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11285.045427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11290.206722] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11304.664774] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11321.828859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11321.839708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11321.849410] Lustre: Skipped 1 previous similar message [11324.293581] Lustre: server umount lustre-MDT0000 complete [11326.946639] LustreError: 359242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11326.969223] LustreError: 359242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11328.380115] LustreError: 358491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977720 with bad export cookie 13674118602011669174 [11328.381795] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11328.398532] LustreError: 358491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11329.035353] Lustre: server umount lustre-MDT0001 complete [11343.997076] Lustre: server umount lustre-OST0000 complete [11361.941274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.784142] LustreError: 362246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11362.949364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11367.910347] LustreError: 362247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.448949] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11376.893651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.245812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11382.347588] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11390.211844] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11395.420812] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11401.272258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11401.498148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11407.418477] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11407.695326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11417.320347] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11433.441655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11433.445942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11433.452305] Lustre: Skipped 3 previous similar messages [11433.454423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11433.461977] Lustre: Skipped 4 previous similar messages [11436.309404] Lustre: server umount lustre-MDT0000 complete [11438.560758] LustreError: 362246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.576443] LustreError: 362246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11440.513671] LustreError: 362228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977832 with bad export cookie 13674118602011670287 [11440.515646] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11440.521628] LustreError: 362228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11440.821265] Lustre: server umount lustre-MDT0001 complete [11457.815596] Lustre: server umount lustre-OST0000 complete [11482.452759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.366591] LustreError: 366157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11483.504668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11489.979776] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11499.350585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11504.872593] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11512.224763] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11518.327154] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11525.346175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11525.750250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11525.755944] Lustre: Skipped 1 previous similar message [11528.843950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11532.539913] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11541.016687] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11549.677940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11549.679810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.689418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11549.695094] LustreError: Skipped 1 previous similar message [11549.735140] Lustre: Skipped 1 previous similar message [11551.566761] Lustre: server umount lustre-OST0000 complete [11555.808450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11555.821865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.849944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11555.862551] Lustre: Skipped 1 previous similar message [11559.909654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11559.916749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11559.928306] Lustre: Skipped 1 previous similar message [11562.010412] Lustre: server umount lustre-MDT0000 complete [11565.031499] LustreError: 366154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.045938] LustreError: 366154:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11565.805491] LustreError: 366137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783977957 with bad export cookie 13674118602011671309 [11565.821917] LustreError: 366137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11565.831319] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11566.091651] Lustre: server umount lustre-MDT0001 complete [11577.682593] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [11580.289933] Key type lgssc unregistered [11580.847759] LNet: 369693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11580.870801] LNetError: 369693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11580.883033] LNet: Removed LNI 192.168.202.114@tcp [11581.945575] Key type .llcrypt unregistered [11581.947267] Key type ._llcrypt unregistered [11608.448940] Key type ._llcrypt registered [11608.450306] Key type .llcrypt registered [11608.521585] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [11609.973710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11610.057746] alg: No test for adler32 (adler32-zlib) [11611.087593] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [11611.300523] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [11612.975351] Key type lgssc registered [11614.275848] Lustre: Echo OBD driver; http://www.lustre.org/ [11623.118822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11623.132944] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11623.136135] LustreError: 371137:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11623.144605] LustreError: 371137:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11623.147048] LustreError: 371137:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11623.149651] LustreError: 371137:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11623.154682] LustreError: 371137:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11630.006298] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11630.008872] LustreError: 371534:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11630.019668] LustreError: 371534:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11630.022971] LustreError: 371534:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11630.026186] LustreError: 371534:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11630.030099] LustreError: 371534:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11648.854918] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [11651.999831] Key type lgssc unregistered [11652.378644] LNet: 372537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11652.384716] LNetError: 372537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11652.395705] LNet: Removed LNI 192.168.202.114@tcp [11653.473485] Key type .llcrypt unregistered [11653.476978] Key type ._llcrypt unregistered [11671.295240] Key type ._llcrypt registered [11671.298836] Key type .llcrypt registered [11671.397222] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [11688.463591] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [11690.010027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11690.205234] alg: No test for adler32 (adler32-zlib) [11691.359599] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [11691.590921] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [11693.233266] Key type lgssc registered [11694.258911] Lustre: Echo OBD driver; http://www.lustre.org/ [11732.870374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11732.891566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.139678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11734.181148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11734.264852] Lustre: lustre-MDT0000: new disk, initializing [11734.411875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11734.438952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11739.295837] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11750.357148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.564394] Lustre: 375697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11750.683955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11750.697492] Lustre: Skipped 1 previous similar message [11750.861577] Lustre: lustre-MDT0001: new disk, initializing [11751.041952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11751.132632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11751.145846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11756.627707] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11766.790899] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11772.355861] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11780.093874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11780.368369] Lustre: lustre-OST0000: new disk, initializing [11780.371605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11780.377941] Lustre: 377666:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11780.463551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11786.851738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11786.865463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11787.020536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11787.324829] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11798.210687] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11802.798431] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11803.062478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11808.520723] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11808.828973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11812.836561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11812.837578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11812.859233] LustreError: Skipped 1 previous similar message [11812.879297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11817.563829] Lustre: server umount lustre-OST0000 complete [11827.684053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11827.713604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11827.744954] Lustre: Skipped 1 previous similar message [11827.773473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11827.791509] Lustre: Skipped 1 previous similar message [11830.585054] Lustre: server umount lustre-MDT0000 complete [11835.777554] LustreError: 377665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978227 with bad export cookie 2347392853505855570 [11835.780197] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11835.784624] LustreError: 377665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11836.022098] Lustre: server umount lustre-MDT0001 complete [11848.112579] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11850.925203] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:30:42 (1783978242) [11859.127314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11859.845628] LustreError: 380127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.869556] LustreError: 380127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11859.976688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11865.062888] LustreError: 380126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.094698] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11869.177909] LustreError: 380127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.273945] LustreError: 380126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.910268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11875.483580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11881.086931] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11892.651771] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11900.505726] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11909.181054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11909.878137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11917.992963] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11928.994108] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11938.795917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11939.054167] Lustre: lustre-OST0001: new disk, initializing [11939.059433] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11939.069199] Lustre: 382969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11939.199263] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11947.672603] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [11949.133972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11949.155435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11949.213896] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11959.140762] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11964.904876] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11964.916296] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11964.923729] LustreError: Skipped 1 previous similar message [11964.924431] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11964.924436] Lustre: Skipped 2 previous similar messages [11964.965963] Lustre: Skipped 2 previous similar messages [11968.663332] Lustre: server umount lustre-OST0001 complete [11970.025430] LustreError: 382621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.059700] LustreError: 382621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11980.261052] LustreError: 381951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.275818] LustreError: 381951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11990.505632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11990.514544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11990.517947] Lustre: Skipped 1 previous similar message [11990.543650] Lustre: Skipped 3 previous similar messages [11993.803611] Lustre: server umount lustre-OST0000 complete [11998.688523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11998.692801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11998.709790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12004.467240] Lustre: server umount lustre-MDT0000 complete [12005.857354] LustreError: 380126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12005.886272] LustreError: 380126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12008.922328] LustreError: 381971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978400 with bad export cookie 2347392853505856452 [12008.924105] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12008.929496] LustreError: 381971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12023.263511] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12023.723345] Lustre: server umount lustre-MDT0001 complete [12037.129121] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12040.288781] Key type lgssc unregistered [12040.645780] LNet: 384957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12040.656137] LNetError: 384957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12040.677179] LNet: Removed LNI 192.168.202.114@tcp [12041.735687] Key type .llcrypt unregistered [12041.739205] Key type ._llcrypt unregistered [12099.476866] Key type ._llcrypt registered [12099.479503] Key type .llcrypt registered [12099.570591] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [12101.253419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12101.280936] alg: No test for adler32 (adler32-zlib) [12102.379786] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [12102.684557] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [12104.407152] Key type lgssc registered [12105.724029] Lustre: Echo OBD driver; http://www.lustre.org/ [12114.179071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12114.208117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12115.573856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12115.613520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12115.809586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12120.484816] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12129.562562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12129.733151] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12129.765918] Lustre: 387758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12130.068827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12134.880214] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12143.246065] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12148.984558] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12155.875776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12155.877083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.882856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12155.946215] Lustre: Skipped 1 previous similar message [12157.434388] Lustre: server umount lustre-MDT0000 complete [12161.394750] LustreError: 387748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978553 with bad export cookie 3585994749482253068 [12161.406236] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12161.877768] Lustre: server umount lustre-MDT0001 complete [12172.510831] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:36:04 (1783978564) [12179.399574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12179.864703] LustreError: 390272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12179.874117] LustreError: 390272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12179.921298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12184.722936] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12185.066742] LustreError: 390273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12190.190576] LustreError: 390272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12193.119328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12193.559317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12198.113405] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12205.857065] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12211.554365] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12218.456241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12218.703061] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12218.896267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12226.326946] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12234.587539] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12241.721459] Lustre: Permanently deactivating lustre-OST0000 [12241.739728] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12250.173790] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12251.700367] LustreError: 393257:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12251.713435] LustreError: 393257:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12251.721472] LustreError: 393257:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12254.885772] LustreError: 393305:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12254.890089] LustreError: 393305:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12263.203617] Lustre: server umount lustre-OST0000 complete [12265.996844] LustreError: 393555:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12266.007652] LustreError: 393555:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12268.516175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12268.526505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12268.533402] Lustre: Skipped 2 previous similar messages [12273.634479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12273.642932] Lustre: Skipped 1 previous similar message [12278.753832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12278.758048] Lustre: Skipped 1 previous similar message [12282.335139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12282.628807] Lustre: server umount lustre-MDT0000 complete [12283.879680] LustreError: 390273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12283.908729] LustreError: 390273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12286.992693] LustreError: 390252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978678 with bad export cookie 3585994749482253649 [12286.999680] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12287.003236] LustreError: 390252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12296.422041] Lustre: server umount lustre-MDT0001 complete [12305.957551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.183679] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12316.708748] LustreError: 394717:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12325.399687] Lustre: 395056:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.114@tcp [12327.659304] Lustre: server umount MGS complete [12342.528586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12343.131527] LustreError: 395781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12343.140289] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12343.167268] Lustre: Skipped 1 previous similar message [12343.267958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12347.988175] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12353.505653] LustreError: 395782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.512699] LustreError: 395782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12356.701209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12357.102818] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12362.437769] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12370.040216] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12376.465824] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12378.742756] Lustre: Permanently reactivating lustre-OST0000 [12378.758770] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12378.764695] Lustre: Skipped 2 previous similar messages [12407.263204] LustreError: 397571:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12414.806589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12415.089852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12415.096651] Lustre: Skipped 1 previous similar message [12421.933172] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12430.775042] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12436.558646] Lustre: server umount lustre-OST0000 complete [12443.106165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12443.114573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12443.130934] Lustre: Skipped 2 previous similar messages [12443.154426] Lustre: Skipped 1 previous similar message [12450.552935] Lustre: server umount lustre-MDT0000 complete [12452.319222] Lustre: 387079:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783978828/real 1783978828] req@00000000cc9d4002 x1870637033424128/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783978844 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12452.358682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12452.380605] LustreError: 395782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12452.400327] LustreError: 395782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12453.408102] Lustre: 387081:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783978829/real 1783978829] req@00000000ae4b0a1c x1870637033424896/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783978845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12455.193941] LustreError: 396516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783978847 with bad export cookie 3585994749482254699 [12455.203947] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12455.212105] LustreError: 396516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12455.582398] Lustre: server umount lustre-MDT0001 complete [12468.277895] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12471.364903] Key type lgssc unregistered [12471.715867] LNet: 400146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12471.724944] LNetError: 400146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12471.763890] LNet: Removed LNI 192.168.202.114@tcp [12472.747425] Key type .llcrypt unregistered [12472.754657] Key type ._llcrypt unregistered [12492.772201] Key type ._llcrypt registered [12492.776326] Key type .llcrypt registered [12492.910899] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [12509.793484] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [12510.989546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12511.253494] alg: No test for adler32 (adler32-zlib) [12512.502698] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [12512.756457] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [12514.497399] Key type lgssc registered [12515.918902] Lustre: Echo OBD driver; http://www.lustre.org/ [12550.798928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12550.822348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12552.134572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12552.167636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12552.263150] Lustre: lustre-MDT0000: new disk, initializing [12552.352774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12552.369160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12556.217785] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12565.797757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12565.920867] Lustre: 403304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12565.982671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12565.990239] Lustre: Skipped 1 previous similar message [12566.048380] Lustre: lustre-MDT0001: new disk, initializing [12566.146431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12566.181083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12566.189043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12570.423088] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12580.250709] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12585.989681] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12592.354785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12592.611153] Lustre: lustre-OST0000: new disk, initializing [12592.615813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12592.625622] Lustre: 405272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12592.710779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12597.941070] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12600.899601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12600.915802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12601.022970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12605.743309] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12610.491247] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12610.839462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12615.506719] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12615.669821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12621.793158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12621.797935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12621.801162] LustreError: Skipped 1 previous similar message [12621.810978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12621.826604] Lustre: Skipped 1 previous similar message [12621.854208] Lustre: Skipped 1 previous similar message [12623.621568] Lustre: server umount lustre-OST0000 complete [12632.032352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12632.035822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12632.052099] Lustre: Skipped 1 previous similar message [12634.277484] Lustre: server umount lustre-MDT0000 complete [12638.255751] LustreError: 403294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979030 with bad export cookie 4212732396116252805 [12638.262905] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12638.265890] LustreError: 403294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12638.608435] Lustre: server umount lustre-MDT0001 complete [12647.310681] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12649.076539] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:44:01 (1783979041) [12655.590708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12655.965160] LustreError: 407732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12655.984963] LustreError: 407732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12656.017199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12660.556386] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12661.237920] LustreError: 407733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12661.249987] LustreError: 407733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12665.318870] LustreError: 407732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12668.546230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12668.984507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12673.603942] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12682.220397] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12689.449727] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12696.599767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12696.971905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12701.928933] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12709.450252] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12711.855160] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12714.427068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12721.632674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12721.646914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12721.662514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12721.669876] Lustre: Skipped 1 previous similar message [12723.687477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12723.690532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12723.693431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12723.717671] Lustre: Skipped 1 previous similar message [12725.507419] Lustre: server umount lustre-OST0000 complete [12730.336678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12730.345135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12730.359714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12735.209992] Lustre: server umount lustre-MDT0000 complete [12738.990663] LustreError: 407712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979130 with bad export cookie 4212732396116253687 [12738.999193] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12739.003604] LustreError: 407712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12739.429375] Lustre: server umount lustre-MDT0001 complete [12750.259369] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12753.809806] Key type lgssc unregistered [12754.112627] LNet: 411456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12754.122087] LNetError: 411456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12754.148692] LNet: Removed LNI 192.168.202.114@tcp [12755.051954] Key type .llcrypt unregistered [12755.060992] Key type ._llcrypt unregistered [12778.680562] Key type ._llcrypt registered [12778.683044] Key type .llcrypt registered [12778.788582] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12779.035439] Key type .llcrypt unregistered [12779.039620] Key type ._llcrypt unregistered [12797.252414] Key type ._llcrypt registered [12797.254898] Key type .llcrypt registered [12797.360858] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [12798.938850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12799.034801] alg: No test for adler32 (adler32-zlib) [12800.089530] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [12800.360367] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [12802.111449] Key type lgssc registered [12803.288615] Lustre: Echo OBD driver; http://www.lustre.org/ [12811.202794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12811.227442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12812.864229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12818.415511] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12823.011528] LustreError: 413670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12823.037219] LustreError: 413670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12827.104354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12827.514567] LustreError: 413671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.553196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12832.738255] LustreError: 413670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12832.758556] LustreError: 413670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12834.259765] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12837.866527] LustreError: 413666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12837.884841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12837.906582] LustreError: 413666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12842.200878] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12842.989887] LustreError: 413666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12843.021083] LustreError: 413666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12849.108447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12849.398778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12854.170979] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12865.003399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12865.005239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12865.024840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12865.045376] Lustre: Skipped 1 previous similar message [12865.061797] Lustre: Skipped 1 previous similar message [12865.964078] Lustre: server umount lustre-OST0000 complete [12870.120724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12870.124836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12870.154260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12871.892851] Lustre: server umount lustre-MDT0000 complete [12875.233526] LustreError: 413666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12875.272411] LustreError: 413666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12875.412094] LustreError: 413652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979267 with bad export cookie 14274019810529256899 [12875.414682] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12875.430857] LustreError: 413652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12876.010436] Lustre: server umount lustre-MDT0001 complete [12887.296638] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [12890.397465] Key type lgssc unregistered [12890.865493] LNet: 416817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12890.879739] LNetError: 416817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12890.929816] LNet: Removed LNI 192.168.202.114@tcp [12892.176413] Key type .llcrypt unregistered [12892.178819] Key type ._llcrypt unregistered [12918.974986] Key type ._llcrypt registered [12918.980804] Key type .llcrypt registered [12919.183676] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [12920.693512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12920.996977] alg: No test for adler32 (adler32-zlib) [12922.116512] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [12922.416310] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [12924.119245] Key type lgssc registered [12925.519969] Lustre: Echo OBD driver; http://www.lustre.org/ [12933.841160] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12933.862244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12949.473313] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12954.592251] LustreError: 418206:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.114@tcp: failed processing log, type 4: rc = -110 [12981.279262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12988.251368] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [12995.959920] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13003.150073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13003.785339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13003.839146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13008.300913] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13008.865869] LustreError: 418232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13008.899464] LustreError: 418232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13013.994064] LustreError: 418231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13016.037714] LustreError: 418232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13016.073561] LustreError: 418232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13016.715795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13017.361309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13021.885580] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13028.689769] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13034.359415] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13042.909965] Lustre: server umount lustre-OST0000 complete [13048.287956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13048.288986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13048.333063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13048.340336] Lustre: Skipped 1 previous similar message [13053.413303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13056.552509] Lustre: server umount lustre-MDT0000 complete [13058.532168] LustreError: 419249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13058.548984] LustreError: 419249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13059.551135] Lustre: 417545:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783979435/real 1783979435] req@00000000423fa0f9 x1870637893185920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783979451 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13059.577545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13059.602262] Lustre: Skipped 1 previous similar message [13060.612069] LustreError: 418213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979452 with bad export cookie 5708245827919732390 [13060.613886] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13060.625341] LustreError: 418213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13061.034232] Lustre: server umount lustre-MDT0001 complete [13071.754591] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [13074.977933] Key type lgssc unregistered [13075.390379] LNet: 421777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13075.395828] LNetError: 421777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13075.413747] LNet: Removed LNI 192.168.202.114@tcp [13076.653441] Key type .llcrypt unregistered [13076.656717] Key type ._llcrypt unregistered [13103.593522] Key type ._llcrypt registered [13103.595149] Key type .llcrypt registered [13103.697822] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [13104.772606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13104.996497] alg: No test for adler32 (adler32-zlib) [13106.063628] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [13106.266890] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [13107.888701] Key type lgssc registered [13109.090694] Lustre: Echo OBD driver; http://www.lustre.org/ [13117.567362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13117.601779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13119.053325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13123.730599] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13128.169565] LustreError: 423197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13128.188761] LustreError: 423197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13132.406880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13133.118514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13138.046891] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13145.591919] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13151.195285] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13158.440628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13158.903555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13164.037442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13165.697972] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13173.274069] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13178.276632] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.281240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13178.294425] Lustre: Skipped 2 previous similar messages [13178.315016] Lustre: Skipped 2 previous similar messages [13180.048946] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.14@tcp (stopping) [13183.470303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13183.636601] Lustre: server umount lustre-MDT0000 complete [13185.157966] LustreError: 423192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13186.191212] LustreError: 423177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.14@tcp arrived at 1783979578 with bad export cookie 6504795734946904552 [13186.213967] LustreError: 423177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13188.447223] LustreError: 423175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783979580 with bad export cookie 6504795734946903663 [13188.464462] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13188.468124] LustreError: 423175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13188.584466] LustreError: 423193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13188.593922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13188.606199] LustreError: 423193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13188.632535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13188.641287] Lustre: Skipped 2 previous similar messages [13193.698149] LustreError: 424199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13193.698946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13193.728790] LustreError: 424199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13194.881603] Lustre: server umount lustre-OST0000 complete [13198.817822] LustreError: 423197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13198.842524] LustreError: 423197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13202.890375] LustreError: 423926:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004be681ab x1870638086126976/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13203.210288] Lustre: server umount lustre-MDT0001 complete [13215.297327] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [13218.219314] Key type lgssc unregistered [13218.680087] LNet: 426881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13218.689685] LNetError: 426881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13218.706917] LNet: Removed LNI 192.168.202.114@tcp [13219.905361] Key type .llcrypt unregistered [13219.907903] Key type ._llcrypt unregistered [13247.271180] Key type ._llcrypt registered [13247.282159] Key type .llcrypt registered [13247.417976] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [13249.123569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13249.182614] alg: No test for adler32 (adler32-zlib) [13250.235567] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [13250.544401] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [13252.303667] Key type lgssc registered [13253.642611] Lustre: Echo OBD driver; http://www.lustre.org/ [13262.700778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13262.735976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13264.462303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13270.042242] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13274.606585] LustreError: 428298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13274.620527] LustreError: 428298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13279.012738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13279.466728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13284.893469] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13294.901851] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13301.454249] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13308.422890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13308.909527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13309.935104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13314.634450] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13323.107151] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13329.388393] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13329.416185] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13329.431403] Lustre: Skipped 1 previous similar message [13331.091679] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [13334.394686] Lustre: server umount lustre-MDT0001 complete [13334.520508] LustreError: 428295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13334.561979] LustreError: 428295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13336.184806] LustreError: 428294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13339.618505] LustreError: 428293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13339.648469] LustreError: 428293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13340.640619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13340.649174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13340.658437] Lustre: Skipped 1 previous similar message [13340.663142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13344.737908] LustreError: 428298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13344.737971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13347.047816] Lustre: server umount lustre-OST0000 complete [13351.314859] Lustre: server umount lustre-MDT0000 complete [13368.944807] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [13372.335114] Key type lgssc unregistered [13372.664075] LNet: 431982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13372.684195] LNetError: 431982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13372.719170] LNet: Removed LNI 192.168.202.114@tcp [13373.868224] Key type .llcrypt unregistered [13373.870442] Key type ._llcrypt unregistered [13401.124220] Key type ._llcrypt registered [13401.127153] Key type .llcrypt registered [13401.237649] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [13417.201899] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [13418.322551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13418.385025] alg: No test for adler32 (adler32-zlib) [13419.739885] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [13419.962298] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [13421.607304] Key type lgssc registered [13422.838426] Lustre: Echo OBD driver; http://www.lustre.org/ [13460.925637] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [13477.437767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13477.472261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13478.775890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13478.845707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13478.999814] Lustre: lustre-MDT0000: new disk, initializing [13479.075744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13479.110235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13483.892381] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13497.762811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13498.140908] Lustre: lustre-OST0000: new disk, initializing [13498.154395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13498.158650] Lustre: Skipped 1 previous similar message [13498.163439] Lustre: 437019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13498.288570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13504.710183] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13508.268993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13508.291425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13508.334662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13513.661485] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13530.186153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13530.374204] Lustre: 436001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13530.470244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13530.602234] Lustre: lustre-MDT0001: new disk, initializing [13530.801589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13530.837821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13530.846498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13536.012649] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13545.634463] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13546.018526] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13551.278793] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13551.653031] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13557.732470] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13557.736627] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13557.737564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13557.744023] Lustre: Skipped 1 previous similar message [13561.488361] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [13561.494115] Lustre: Skipped 1 previous similar message [13562.857076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13563.059914] Lustre: server umount lustre-MDT0001 complete [13573.091099] LustreError: 436011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13573.103235] LustreError: 436011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13578.217111] LustreError: 439587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13578.231895] LustreError: 439587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13580.256664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13580.262049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13580.272146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13580.275961] Lustre: Skipped 1 previous similar message [13583.329461] LustreError: 436011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13586.154496] Lustre: server umount lustre-OST0000 complete [13588.462499] LustreError: 439587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13603.295369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13603.590719] Lustre: server umount lustre-MDT0000 complete [13611.375211] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [13612.920639] Key type lgssc unregistered [13613.092382] LNet: 440766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13613.095446] LNetError: 440766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13613.102650] LNet: Removed LNI 192.168.202.114@tcp [13613.551776] Key type .llcrypt unregistered [13613.552746] Key type ._llcrypt unregistered [13711.506805] Key type ._llcrypt registered [13711.509904] Key type .llcrypt registered [13711.580408] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [13711.748702] Key type .llcrypt unregistered [13711.750364] Key type ._llcrypt unregistered [13722.949568] Key type ._llcrypt registered [13722.950955] Key type .llcrypt registered [13723.022377] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [13723.718083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13723.733096] alg: No test for adler32 (adler32-zlib) [13724.723755] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [13724.903036] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [13726.559335] Key type lgssc registered [13727.486956] Lustre: Echo OBD driver; http://www.lustre.org/ [13742.374808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13742.384949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13743.499339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13743.510517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13743.548945] Lustre: lustre-MDT0000: new disk, initializing [13743.619265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13743.628638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13745.632182] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13750.390611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13750.438013] Lustre: 444191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13750.471344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13750.472399] Lustre: Skipped 1 previous similar message [13750.509570] Lustre: lustre-MDT0001: new disk, initializing [13750.554517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13750.570523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13750.575785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13752.210461] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13759.546356] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13764.447404] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13770.329637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13770.522953] Lustre: lustre-OST0000: new disk, initializing [13770.526475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13770.532346] Lustre: 446157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13770.596604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13775.653430] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13780.515719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13780.524864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13780.669412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13783.311712] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13791.203692] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13791.219988] Lustre: Skipped 1 previous similar message [13791.220633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13791.234019] Lustre: Skipped 2 previous similar messages [13796.758628] Lustre: server umount lustre-MDT0000 complete [13800.759053] LustreError: 444184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980192 with bad export cookie 12756126625843421372 [13800.765404] LustreError: 444184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13800.766709] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13801.043456] Lustre: server umount lustre-MDT0001 complete [13814.972362] Lustre: server umount lustre-OST0000 complete [13859.681713] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:04:10 (1783980250) [13868.117585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13868.565679] LustreError: 448205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13868.593597] LustreError: 448205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13868.655867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13873.762961] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13874.148589] LustreError: 448206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13878.243539] LustreError: 448205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13883.030774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13883.332447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13888.192774] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13895.504353] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13901.586285] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13921.093961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13921.376052] Lustre: Found index 0 for lustre-OST0000, updating log [13921.566325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13928.083921] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [13935.418621] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13940.033977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13945.320171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13945.329827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13945.335356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13945.366955] Lustre: Skipped 2 previous similar messages [13946.468305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13946.475069] Lustre: Skipped 1 previous similar message [13946.905972] Lustre: server umount lustre-OST0000 complete [13962.218749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13962.223290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13962.224439] Lustre: Skipped 1 previous similar message [13962.232496] Lustre: Skipped 1 previous similar message [13967.328175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13967.348196] Lustre: Skipped 1 previous similar message [13968.798662] Lustre: server umount lustre-MDT0000 complete [13972.454180] LustreError: 448200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13972.473441] LustreError: 448200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13972.876851] LustreError: 448186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783980364 with bad export cookie 12756126625843422751 [13972.883847] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13972.887214] LustreError: 448186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13973.166548] Lustre: server umount lustre-MDT0001 complete [13982.324317] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:06:14 (1783980374) [13988.063330] LNet: 1 local NIs in recovery (showing 1): 192.168.202.114@tcp [14604.258761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14604.885708] LustreError: 454619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14604.944895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14610.412718] LustreError: 454620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14611.182700] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [14614.498695] LustreError: 454619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14620.655075] LustreError: 454619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14620.676261] LustreError: 454619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14620.873666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14621.362029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14626.722893] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [14635.343568] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14641.097068] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14660.939738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14661.321877] Lustre: Found index 0 for lustre-OST0000, updating log [14661.826106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14666.976783] LNet: 1 local NIs in recovery (showing 1): 192.168.202.114@tcp [14669.126405] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [14678.238729] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14680.557742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14680.566273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14680.579085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14694.367213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14694.663856] Lustre: server umount lustre-OST0000 complete [14698.465961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14698.483755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14698.513459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14699.553215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14700.576894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14700.590839] Lustre: Skipped 3 previous similar messages [14704.673372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14704.681932] Lustre: Skipped 6 previous similar messages [14705.147450] Lustre: server umount lustre-MDT0000 complete [14705.699891] LustreError: 454619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14705.723087] LustreError: 454619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14709.260296] LustreError: 454599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783981101 with bad export cookie 12756126625843423836 [14709.270579] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14709.275140] LustreError: 454599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14709.871563] Lustre: server umount lustre-MDT0001 complete [14726.368775] LNet: 1 local NIs in recovery (showing 1): 192.168.202.114@tcp [14740.754151] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [14753.375575] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [14791.080790] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:19:42 (1783981182) [14810.798639] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [14813.595270] Key type lgssc unregistered [14813.874558] LNet: 461237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14813.881147] LNetError: 461237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14923.506311] LNet: Removed LNI 192.168.202.114@tcp [14924.501170] Key type .llcrypt unregistered [14924.503062] Key type ._llcrypt unregistered [14942.341569] Key type ._llcrypt registered [14942.344672] Key type .llcrypt registered [14942.443742] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [14943.564625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14943.662499] alg: No test for adler32 (adler32-zlib) [14944.670894] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [14944.883756] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [14946.511189] Key type lgssc registered [14947.758558] Lustre: Echo OBD driver; http://www.lustre.org/ [14951.358697] LNet: Added LNI 192.168.202.114@tcp100 [8/256/0/180] [14952.181829] LNet: Added LNI 192.168.202.114@tcp101 [8/256/0/180] [14952.978722] LNet: Added LNI 192.168.202.114@tcp102 [8/256/0/180] [14953.700090] LNet: Added LNI 192.168.202.114@tcp103 [8/256/0/180] [14954.434027] LNet: Added LNI 192.168.202.114@tcp104 [8/256/0/180] [14955.161299] LNet: Added LNI 192.168.202.114@tcp105 [8/256/0/180] [14956.087705] LNet: Added LNI 192.168.202.114@tcp106 [8/256/0/180] [14957.014623] LNet: Added LNI 192.168.202.114@tcp107 [8/256/0/180] [14957.889665] LNet: Added LNI 192.168.202.114@tcp108 [8/256/0/180] [14958.740340] LNet: Added LNI 192.168.202.114@tcp109 [8/256/0/180] [14959.470600] LNet: Added LNI 192.168.202.114@tcp110 [8/256/0/180] [14960.143734] LNet: Added LNI 192.168.202.114@tcp111 [8/256/0/180] [14961.081312] LNet: Added LNI 192.168.202.114@tcp112 [8/256/0/180] [14962.015383] LNet: Added LNI 192.168.202.114@tcp113 [8/256/0/180] [14962.937570] LNet: Added LNI 192.168.202.114@tcp114 [8/256/0/180] [14963.992670] LNet: Added LNI 192.168.202.114@tcp115 [8/256/0/180] [14965.078739] LNet: Added LNI 192.168.202.114@tcp116 [8/256/0/180] [14966.044727] LNet: Added LNI 192.168.202.114@tcp117 [8/256/0/180] [14966.972634] LNet: Added LNI 192.168.202.114@tcp118 [8/256/0/180] [14967.839740] LNet: Added LNI 192.168.202.114@tcp119 [8/256/0/180] [14968.611948] LNet: Added LNI 192.168.202.114@tcp120 [8/256/0/180] [14969.426634] LNet: Added LNI 192.168.202.114@tcp121 [8/256/0/180] [14970.345800] LNet: Added LNI 192.168.202.114@tcp122 [8/256/0/180] [14971.355875] LNet: Added LNI 192.168.202.114@tcp123 [8/256/0/180] [14972.166616] LNet: Added LNI 192.168.202.114@tcp124 [8/256/0/180] [14972.928038] LNet: Added LNI 192.168.202.114@tcp125 [8/256/0/180] [14973.849992] LNet: Added LNI 192.168.202.114@tcp126 [8/256/0/180] [14974.629514] LNet: Added LNI 192.168.202.114@tcp127 [8/256/0/180] [14975.489286] LNet: Added LNI 192.168.202.114@tcp128 [8/256/0/180] [14976.377599] LNet: Added LNI 192.168.202.114@tcp129 [8/256/0/180] [14977.171704] LNet: Added LNI 192.168.202.114@tcp130 [8/256/0/180] [14977.912447] LNet: Added LNI 192.168.202.114@tcp131 [8/256/0/180] [14978.613314] LNet: Added LNI 192.168.202.114@tcp132 [8/256/0/180] [14979.396919] LNet: Added LNI 192.168.202.114@tcp133 [8/256/0/180] [14980.239496] LNet: Added LNI 192.168.202.114@tcp134 [8/256/0/180] [14981.067500] LNet: Added LNI 192.168.202.114@tcp135 [8/256/0/180] [14987.786913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14987.817902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14988.938453] Lustre: 464269:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.114@tcp: can't add NID 192.168.202.114@tcp130: rc = -75 [14989.015230] LNetError: 461727:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.114 [14989.038211] LNetError: Protocol error connecting to 192.168.202.114@tcp115 at host 192.168.202.114:988: is it running a compatible version of Lustre? [14989.055248] LNetError: Skipped 8 previous similar messages [14989.175151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14989.194678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14989.266558] Lustre: lustre-MDT0000: new disk, initializing [14989.328130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14989.348238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14989.990222] LNet: 6 local NIs in recovery (showing 6): 192.168.202.114@tcp124, 192.168.202.114@tcp126, 192.168.202.114@tcp127, 192.168.202.114@tcp125, 192.168.202.114@tcp129, 192.168.202.114@tcp128 [14990.112100] LNetError: 464302:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.114 [14990.139569] LNetError: 464302:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [14990.154535] LNetError: Protocol error connecting to 192.168.202.114@tcp116 at host 192.168.202.114:988: is it running a compatible version of Lustre? [14990.170836] LNetError: Skipped 1 previous similar message [14993.128560] LNetError: 461727:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.114 [14993.142329] LNetError: 461727:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [14993.184734] LNetError: Protocol error connecting to 192.168.202.114@tcp124 at host 192.168.202.114:988: is it running a compatible version of Lustre? [14993.200537] LNetError: Skipped 7 previous similar messages [14994.197596] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [14997.224634] LNetError: 464295:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.114 [14997.246163] LNetError: 464295:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14997.255367] LNetError: Protocol error connecting to 192.168.202.114@tcp124 at host 192.168.202.114:988: is it running a compatible version of Lustre? [15005.166365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15005.324228] Lustre: 464284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15005.382951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15005.386546] Lustre: Skipped 1 previous similar message [15005.409215] LNetError: 461726:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.114 [15005.422620] LNetError: Protocol error connecting to 192.168.202.114@tcp124 at host 192.168.202.114:988: is it running a compatible version of Lustre? [15005.484236] Lustre: lustre-MDT0001: new disk, initializing [15005.521657] Lustre: 465086:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.114@tcp: can't add NID 192.168.202.114@tcp130: rc = -75 [15005.530451] Lustre: 465086:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15005.638297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15005.664129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15005.679249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15007.720455] Lustre: 465209:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.114@tcp: can't add NID 192.168.202.114@tcp130: rc = -75 [15007.730782] Lustre: 465209:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15009.704377] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15017.617855] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15020.645662] LNetError: 464291:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.114 [15020.658096] LNetError: Protocol error connecting to 192.168.202.114@tcp122 at host 192.168.202.114:988: is it running a compatible version of Lustre? [15022.364615] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15029.546683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15029.882179] Lustre: lustre-OST0000: new disk, initializing [15029.894758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15029.902825] Lustre: 466260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15029.986720] Lustre: 466265:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.114@tcp: can't add NID 192.168.202.114@tcp130: rc = -75 [15029.994790] Lustre: 466265:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15030.053461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15036.317561] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15037.024935] LNetError: 464302:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.114 [15037.035272] LNetError: 464302:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [15037.048464] LNetError: Protocol error connecting to 192.168.202.114@tcp123 at host 192.168.202.114:988: is it running a compatible version of Lustre? [15037.066772] LNetError: Skipped 11 previous similar messages [15040.049910] Lustre: 466751:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.114@tcp: can't add NID 192.168.202.114@tcp130: rc = -75 [15040.057962] Lustre: 466751:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15040.123641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15040.141617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15040.285267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15045.573933] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15049.442975] LNet: 1 local NIs in recovery (showing 1): 192.168.202.114@tcp124 [15055.861034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15055.861411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15055.891593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15055.905159] Lustre: Skipped 1 previous similar message [15058.676239] Lustre: server umount lustre-OST0000 complete [15066.081688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15066.086405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15066.100829] Lustre: Skipped 2 previous similar messages [15068.891992] Lustre: server umount lustre-MDT0000 complete [15072.714399] LustreError: 464278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783981464 with bad export cookie 12452027834321456085 [15072.716703] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15072.728666] LustreError: 464278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15073.014596] Lustre: server umount lustre-MDT0001 complete [15080.055761] LNetError: 464299:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.114 [15080.072463] LNetError: 464299:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 20 previous similar messages [15080.100811] LNetError: Protocol error connecting to 192.168.202.114@tcp116 at host 192.168.202.114:988: is it running a compatible version of Lustre? [15080.117193] LNetError: Skipped 20 previous similar messages [15085.220590] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [15088.306802] Key type lgssc unregistered [15088.702569] LNet: 468174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15088.711454] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15088.716677] LNet: Removed LNI 192.168.202.114@tcp [15089.759163] LNet: Removed LNI 192.168.202.114@tcp100 [15089.773491] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15089.783367] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15089.789386] LNet: Removed LNI 192.168.202.114@tcp101 [15090.847474] LNet: Removed LNI 192.168.202.114@tcp102 [15090.874845] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15090.890305] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15090.907289] LNet: Removed LNI 192.168.202.114@tcp103 [15091.937641] LNet: Removed LNI 192.168.202.114@tcp104 [15092.960905] LNet: Removed LNI 192.168.202.114@tcp105 [15092.965677] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15092.973105] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15092.987064] LNet: Removed LNI 192.168.202.114@tcp106 [15094.047318] LNet: Removed LNI 192.168.202.114@tcp107 [15095.071710] LNet: Removed LNI 192.168.202.114@tcp108 [15096.095183] LNet: Removed LNI 192.168.202.114@tcp109 [15097.119182] LNet: Removed LNI 192.168.202.114@tcp110 [15097.128605] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15097.147185] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15097.165269] LNet: Removed LNI 192.168.202.114@tcp111 [15098.207551] LNet: Removed LNI 192.168.202.114@tcp112 [15099.231503] LNet: Removed LNI 192.168.202.114@tcp113 [15100.255388] LNet: Removed LNI 192.168.202.114@tcp114 [15101.280142] LNet: Removed LNI 192.168.202.114@tcp115 [15102.304981] LNet: Removed LNI 192.168.202.114@tcp116 [15103.327176] LNet: Removed LNI 192.168.202.114@tcp117 [15104.353056] LNet: Removed LNI 192.168.202.114@tcp118 [15105.377199] LNet: Removed LNI 192.168.202.114@tcp119 [15105.387872] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15105.394503] LNetError: 468174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15105.414216] LNet: Removed LNI 192.168.202.114@tcp120 [15106.463871] LNet: Removed LNI 192.168.202.114@tcp121 [15107.487901] LNet: Removed LNI 192.168.202.114@tcp122 [15108.511258] LNet: Removed LNI 192.168.202.114@tcp123 [15108.514141] LNet: Removed LNI 192.168.202.114@tcp124 [15109.535144] LNet: Removed LNI 192.168.202.114@tcp125 [15110.559123] LNet: Removed LNI 192.168.202.114@tcp126 [15111.647460] LNet: Removed LNI 192.168.202.114@tcp127 [15112.672282] LNet: Removed LNI 192.168.202.114@tcp128 [15113.695157] LNet: Removed LNI 192.168.202.114@tcp129 [15113.697485] LNet: Removed LNI 192.168.202.114@tcp130 [15113.704333] LNet: Removed LNI 192.168.202.114@tcp131 [15113.709435] LNet: Removed LNI 192.168.202.114@tcp132 [15113.712668] LNet: Removed LNI 192.168.202.114@tcp133 [15113.715167] LNet: Removed LNI 192.168.202.114@tcp134 [15113.728619] LNet: Removed LNI 192.168.202.114@tcp135 [15114.626585] Key type .llcrypt unregistered [15114.628786] Key type ._llcrypt unregistered [15137.250718] Key type ._llcrypt registered [15137.252981] Key type .llcrypt registered [15137.330659] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [15153.515963] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [15154.942584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15154.990078] alg: No test for adler32 (adler32-zlib) [15156.098592] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [15156.386531] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [15158.127184] Key type lgssc registered [15159.408473] Lustre: Echo OBD driver; http://www.lustre.org/ [15191.904869] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:26:23 (1783981583) [15208.749664] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [15211.031524] Key type lgssc unregistered [15211.311874] LNet: 472262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15211.316713] LNetError: 472262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15211.328162] LNet: Removed LNI 192.168.202.114@tcp [15212.075626] Key type .llcrypt unregistered [15212.077540] Key type ._llcrypt unregistered [15227.560963] Key type ._llcrypt registered [15227.563129] Key type .llcrypt registered [15227.655723] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [15228.766420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15228.785048] alg: No test for adler32 (adler32-zlib) [15229.839808] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [15230.075392] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [15231.735205] Key type lgssc registered [15232.957993] Lustre: Echo OBD driver; http://www.lustre.org/ [15240.761349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15240.801194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15242.085029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15242.124501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15242.203329] Lustre: lustre-MDT0000: new disk, initializing [15242.266131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15242.289032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15246.838602] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15256.472426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15256.575297] Lustre: 473473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15256.624938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15256.628796] Lustre: Skipped 1 previous similar message [15256.691624] Lustre: lustre-MDT0001: new disk, initializing [15256.767593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15256.805527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15256.818865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15261.155348] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15269.628632] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15274.318509] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15280.674481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15281.015472] Lustre: lustre-OST0000: new disk, initializing [15281.027825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15281.037633] Lustre: 475443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15281.111593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15286.360054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15286.374179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15286.455644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15286.666160] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15295.853995] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15300.522418] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing lnet_if_list [15305.917946] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15305.946176] LNet: Added LNI 192.168.202.114@tcp10 [8/256/0/180] [15311.732732] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15311.761892] LNet: Added LNI 192.168.202.114@tcp11 [8/256/0/180] [15315.754484] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15315.780800] LNet: Added LNI 192.168.202.114@tcp12 [8/256/0/180] [15318.902531] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15318.920035] LNet: Added LNI 192.168.202.114@tcp13 [8/256/0/180] [15323.149559] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15323.170192] LNet: Added LNI 192.168.202.114@tcp14 [8/256/0/180] [15328.094780] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15328.122058] LNet: Added LNI 192.168.202.114@tcp15 [8/256/0/180] [15332.871906] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15332.900071] LNet: Added LNI 192.168.202.114@tcp16 [8/256/0/180] [15337.451707] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15337.477088] LNet: Added LNI 192.168.202.114@tcp17 [8/256/0/180] [15342.650762] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15342.677626] LNet: Added LNI 192.168.202.114@tcp18 [8/256/0/180] [15347.641035] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15347.672501] LNet: Added LNI 192.168.202.114@tcp19 [8/256/0/180] [15352.562425] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15352.581353] LNet: Added LNI 192.168.202.114@tcp20 [8/256/0/180] [15357.681155] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15357.724966] LNet: Added LNI 192.168.202.114@tcp21 [8/256/0/180] [15362.764326] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15362.825543] LNet: Added LNI 192.168.202.114@tcp22 [8/256/0/180] [15368.514764] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15368.537763] LNet: Added LNI 192.168.202.114@tcp23 [8/256/0/180] [15373.340906] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15373.367618] LNet: Added LNI 192.168.202.114@tcp24 [8/256/0/180] [15378.532425] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15378.552181] LNet: Added LNI 192.168.202.114@tcp25 [8/256/0/180] [15383.877582] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15383.903087] LNet: Added LNI 192.168.202.114@tcp26 [8/256/0/180] [15389.198660] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15389.236718] LNet: Added LNI 192.168.202.114@tcp27 [8/256/0/180] [15395.008381] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15395.029831] LNet: Added LNI 192.168.202.114@tcp28 [8/256/0/180] [15400.895039] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15400.920236] LNet: Added LNI 192.168.202.114@tcp29 [8/256/0/180] [15406.526677] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15406.555216] LNet: Added LNI 192.168.202.114@tcp30 [8/256/0/180] [15412.407907] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15412.431491] LNet: Added LNI 192.168.202.114@tcp31 [8/256/0/180] [15418.190594] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15418.215682] LNet: Added LNI 192.168.202.114@tcp32 [8/256/0/180] [15424.745410] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15424.772474] LNet: Added LNI 192.168.202.114@tcp33 [8/256/0/180] [15432.060994] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15432.113372] LNet: Added LNI 192.168.202.114@tcp34 [8/256/0/180] [15439.722324] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15439.747370] LNet: Added LNI 192.168.202.114@tcp35 [8/256/0/180] [15447.121382] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15447.163871] LNet: Added LNI 192.168.202.114@tcp36 [8/256/0/180] [15453.460296] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15453.476223] LNet: Added LNI 192.168.202.114@tcp37 [8/256/0/180] [15459.608552] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15459.635608] LNet: Added LNI 192.168.202.114@tcp38 [8/256/0/180] [15466.695916] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15466.720433] LNet: Added LNI 192.168.202.114@tcp39 [8/256/0/180] [15473.148338] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15473.171874] LNet: Added LNI 192.168.202.114@tcp40 [8/256/0/180] [15480.804798] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15480.845931] LNet: Added LNI 192.168.202.114@tcp41 [8/256/0/180] [15487.645397] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15487.675484] LNet: Added LNI 192.168.202.114@tcp42 [8/256/0/180] [15495.451043] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15495.487039] LNet: Added LNI 192.168.202.114@tcp43 [8/256/0/180] [15503.550620] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15503.581658] LNet: Added LNI 192.168.202.114@tcp44 [8/256/0/180] [15511.130344] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15511.155957] LNet: Added LNI 192.168.202.114@tcp45 [8/256/0/180] [15517.722385] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15517.737392] LNet: Added LNI 192.168.202.114@tcp46 [8/256/0/180] [15525.242616] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15525.268657] LNet: Added LNI 192.168.202.114@tcp47 [8/256/0/180] [15533.013401] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15533.031275] LNet: Added LNI 192.168.202.114@tcp48 [8/256/0/180] [15541.080655] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15541.109371] LNet: Added LNI 192.168.202.114@tcp49 [8/256/0/180] [15550.456384] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15550.477970] LNet: Added LNI 192.168.202.114@tcp50 [8/256/0/180] [15569.392520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15569.411539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15569.453226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15571.435533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15571.436829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15571.452867] Lustre: Skipped 1 previous similar message [15571.465718] Lustre: Skipped 2 previous similar messages [15573.870914] Lustre: server umount lustre-MDT0000 complete [15577.342333] LustreError: 473466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783981969 with bad export cookie 754769414803893718 [15577.344828] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15577.352859] LustreError: 473466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15577.585709] Lustre: server umount lustre-MDT0001 complete [15594.788783] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [15598.048404] Lustre: 472747:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783981973/real 1783981973] req@0000000063ab60e3 x1870640312387328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783981989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15598.079465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15599.072513] Lustre: 472744:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783981974/real 1783981974] req@00000000aab9fe6f x1870640312387456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783981990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15602.055645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15602.144531] Lustre: 472745:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783981978/real 1783981978] req@0000000035cf34a3 x1870640312387584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783981994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15602.460250] LustreError: 475436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15602.480649] LustreError: 475436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15602.527308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15603.552753] LustreError: 475437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15606.560726] LustreError: 475436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15606.883192] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15608.615980] Lustre: 472746:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783981984/real 1783981984] req@00000000823f4928 x1870640312387968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783982000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15608.661687] Lustre: 472746:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15612.913323] LustreError: 475437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15612.931219] LustreError: 475437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15613.927708] Lustre: 472744:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783981989/real 1783981989] req@00000000e6511529 x1870640312388096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783982005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15615.378546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15615.862437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15620.117179] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15621.099812] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15626.226089] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15626.937246] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15631.501744] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15651.811224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15651.814194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15651.818592] Lustre: Skipped 1 previous similar message [15651.818924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15651.821676] Lustre: Skipped 1 previous similar message [15655.908803] Lustre: server umount lustre-OST0000 complete [15662.049870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15662.051431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15662.062210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15662.070466] Lustre: Skipped 1 previous similar message [15662.080365] Lustre: Skipped 1 previous similar message [15665.763772] Lustre: server umount lustre-MDT0000 complete [15667.170504] LustreError: 499682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15667.199115] LustreError: 499682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15668.964284] LustreError: 479609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783982060 with bad export cookie 754769414803896238 [15668.968528] LustreError: 479609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15668.972330] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15669.325774] Lustre: server umount lustre-MDT0001 complete [15678.441567] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [15680.849440] Key type lgssc unregistered [15681.227235] LNet: 502480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15681.234639] LNetError: 502480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15681.240938] LNet: Removed LNI 192.168.202.114@tcp [15681.243631] LNet: Removed LNI 192.168.202.114@tcp10 [15681.245543] LNet: Removed LNI 192.168.202.114@tcp11 [15681.247779] LNet: Removed LNI 192.168.202.114@tcp12 [15681.250198] LNet: Removed LNI 192.168.202.114@tcp13 [15681.252252] LNet: Removed LNI 192.168.202.114@tcp14 [15681.256288] LNet: Removed LNI 192.168.202.114@tcp15 [15681.259695] LNet: Removed LNI 192.168.202.114@tcp16 [15681.264400] LNet: Removed LNI 192.168.202.114@tcp17 [15681.266828] LNet: Removed LNI 192.168.202.114@tcp18 [15681.270413] LNet: Removed LNI 192.168.202.114@tcp19 [15681.272711] LNet: Removed LNI 192.168.202.114@tcp20 [15681.275864] LNet: Removed LNI 192.168.202.114@tcp21 [15681.278856] LNet: Removed LNI 192.168.202.114@tcp22 [15681.281563] LNet: Removed LNI 192.168.202.114@tcp23 [15681.285415] LNet: Removed LNI 192.168.202.114@tcp24 [15681.288132] LNet: Removed LNI 192.168.202.114@tcp25 [15681.291777] LNet: Removed LNI 192.168.202.114@tcp26 [15681.294511] LNet: Removed LNI 192.168.202.114@tcp27 [15681.297303] LNet: Removed LNI 192.168.202.114@tcp28 [15681.300212] LNet: Removed LNI 192.168.202.114@tcp29 [15681.303268] LNet: Removed LNI 192.168.202.114@tcp30 [15681.306171] LNet: Removed LNI 192.168.202.114@tcp31 [15681.310628] LNet: Removed LNI 192.168.202.114@tcp32 [15681.317192] LNet: Removed LNI 192.168.202.114@tcp33 [15681.321503] LNet: Removed LNI 192.168.202.114@tcp34 [15681.323334] LNet: Removed LNI 192.168.202.114@tcp35 [15681.325126] LNet: Removed LNI 192.168.202.114@tcp36 [15681.328331] LNet: Removed LNI 192.168.202.114@tcp37 [15681.330729] LNet: Removed LNI 192.168.202.114@tcp38 [15681.333316] LNet: Removed LNI 192.168.202.114@tcp39 [15681.336451] LNet: Removed LNI 192.168.202.114@tcp40 [15681.339897] LNet: Removed LNI 192.168.202.114@tcp41 [15681.342704] LNet: Removed LNI 192.168.202.114@tcp42 [15681.345174] LNet: Removed LNI 192.168.202.114@tcp43 [15681.347781] LNet: Removed LNI 192.168.202.114@tcp44 [15681.350544] LNet: Removed LNI 192.168.202.114@tcp45 [15681.353868] LNet: Removed LNI 192.168.202.114@tcp46 [15681.357143] LNet: Removed LNI 192.168.202.114@tcp47 [15681.359828] LNet: Removed LNI 192.168.202.114@tcp48 [15681.363947] LNet: Removed LNI 192.168.202.114@tcp49 [15681.376622] LNet: Removed LNI 192.168.202.114@tcp50 [15682.158940] Key type .llcrypt unregistered [15682.161602] Key type ._llcrypt unregistered [15701.381370] Key type ._llcrypt registered [15701.384664] Key type .llcrypt registered [15701.495812] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [15713.947297] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [15715.247503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15715.302107] alg: No test for adler32 (adler32-zlib) [15716.430131] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [15716.662803] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [15718.311249] Key type lgssc registered [15719.674529] Lustre: Echo OBD driver; http://www.lustre.org/ [15749.162052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15749.188920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15750.479269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15750.507618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15750.567554] Lustre: lustre-MDT0000: new disk, initializing [15750.630244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15750.643345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15754.694741] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15763.608068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15763.695524] Lustre: 505656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15763.735536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15763.738788] Lustre: Skipped 1 previous similar message [15763.788981] Lustre: lustre-MDT0001: new disk, initializing [15763.856583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15763.889930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15763.903697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15767.656644] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15774.758983] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15779.100708] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15785.473713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15785.640586] Lustre: *** cfs_fail_loc=90f, val=0*** [15785.700564] Lustre: lustre-OST0000: new disk, initializing [15785.704976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15785.713240] Lustre: 507669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15785.786515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15791.451486] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15792.742912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15792.760921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15792.870464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15800.226489] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15808.488252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15808.488365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15808.497309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15808.497450] LustreError: Skipped 1 previous similar message [15808.526713] Lustre: Skipped 1 previous similar message [15812.798864] Lustre: server umount lustre-OST0000 complete [15818.730889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15818.747685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15818.751463] Lustre: Skipped 1 previous similar message [15822.466790] Lustre: server umount lustre-MDT0000 complete [15826.348770] LustreError: 505648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783982218 with bad export cookie 10006725757174135182 [15826.368236] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15826.371116] LustreError: 505648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15826.808761] Lustre: server umount lustre-MDT0001 complete [15836.811484] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [15839.040354] Key type lgssc unregistered [15839.252428] LNet: 509575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15839.263209] LNetError: 509575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15839.282709] LNet: Removed LNI 192.168.202.114@tcp [15840.035736] Key type .llcrypt unregistered [15840.039580] Key type ._llcrypt unregistered [15857.694526] Key type ._llcrypt registered [15857.696851] Key type .llcrypt registered [15857.774665] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [15858.978813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15859.020612] alg: No test for adler32 (adler32-zlib) [15860.077725] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [15860.308660] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [15862.015194] Key type lgssc registered [15862.891055] Lustre: Echo OBD driver; http://www.lustre.org/ [15868.878375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15868.893165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15870.252968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15873.730878] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15880.675135] LustreError: 510845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15880.695954] LustreError: 510845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15880.735821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15881.004148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15885.102490] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15891.741229] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15896.301488] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15901.926357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15902.213875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15907.522854] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15914.154155] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15923.953163] loop0: detected capacity change from 0 to 204800000 [15924.104339] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15924.113825] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15924.121753] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15929.586424] loop0: detected capacity change from 0 to 204800000 [15929.747644] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15929.755562] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15929.765183] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15933.765630] loop0: detected capacity change from 0 to 204800000 [15938.098733] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15938.189935] Lustre: fs15246-OST0000: new disk, initializing [15938.194373] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15938.200636] Lustre: 514436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15938.256246] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15943.758121] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15951.931153] loop1: detected capacity change from 0 to 204800000 [15956.766811] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15956.892537] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15956.975980] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15957.060589] Lustre: fs15246-MDT0000: new disk, initializing [15957.357746] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15957.412931] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15957.506386] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15958.561785] Lustre: 515560:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000010520fd1 x1870640973915520/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783982366 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15961.317392] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [15967.652987] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15967.672980] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15967.698499] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15980.017440] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15980.034539] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15985.127346] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15985.642168] Lustre: server umount fs15246-MDT0000 complete [15990.251286] LustreError: 510839:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15993.627647] Lustre: server umount fs15246-OST0000 complete [16005.620633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16005.632794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16005.640710] Lustre: Skipped 1 previous similar message [16005.656404] Lustre: Skipped 1 previous similar message [16010.606674] Lustre: server umount lustre-OST0000 complete [16015.844897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16015.852611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16015.859278] Lustre: Skipped 1 previous similar message [16020.970231] LustreError: 510841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16020.979569] LustreError: 510841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16021.013518] Lustre: server umount lustre-MDT0000 complete [16024.619707] LustreError: 516698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783982416 with bad export cookie 8993798910729729705 [16024.622501] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16024.626145] LustreError: 516698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16024.906219] Lustre: server umount lustre-MDT0001 complete [16035.333599] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [16038.375855] Key type lgssc unregistered [16038.730894] LNet: 518659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16038.741723] LNetError: 518659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16038.753311] LNet: Removed LNI 192.168.202.114@tcp [16039.873754] Key type .llcrypt unregistered [16039.882310] Key type ._llcrypt unregistered [16056.837223] Key type ._llcrypt registered [16056.842160] Key type .llcrypt registered [16056.957476] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [16058.482397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16058.494039] alg: No test for adler32 (adler32-zlib) [16059.473658] Lustre: Lustre: Build Version: 2.17.54_162_gc83a79c [16059.658579] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [16061.280157] Key type lgssc registered [16062.127727] Lustre: Echo OBD driver; http://www.lustre.org/ [16094.152760] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [16104.018856] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [16137.709952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16137.737404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16139.078248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16139.102173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16139.198916] Lustre: lustre-MDT0000: new disk, initializing [16139.273386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16139.291667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16143.673820] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [16153.622307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16153.784509] Lustre: 523123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16153.834596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16153.839290] Lustre: Skipped 1 previous similar message [16153.925952] Lustre: lustre-MDT0001: new disk, initializing [16153.999903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16154.030017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16154.047295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16158.567590] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [16167.022506] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16171.775591] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16177.767233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16178.033600] Lustre: lustre-OST0000: new disk, initializing [16178.038482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16178.046766] Lustre: 525092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16178.138447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16184.280817] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all [16185.384206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16185.393400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16185.502347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16192.237468] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16196.724670] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16196.956506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16200.323417] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16200.480752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16205.796411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16205.808767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16205.832956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16206.312775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16206.329598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16206.339397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16208.272694] Lustre: server umount lustre-OST0000 complete [16215.525306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16215.544256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16215.561779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16215.568371] Lustre: Skipped 1 previous similar message [16221.669331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16221.687098] Lustre: Skipped 3 previous similar messages [16226.272048] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16226.479510] Lustre: server umount lustre-MDT0000 complete [16229.367755] LustreError: 523117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783982621 with bad export cookie 4415468281093043852 [16229.373463] LustreError: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16229.381633] LustreError: 523117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16229.753707] Lustre: server umount lustre-MDT0001 complete [16248.976703] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [16256.913809] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [16279.853735] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15967 sec ========= 18:44:32 (1783982672) [16281.159822] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:44:33 (1783982673) === [16283.429603] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:44:35 (1783982675) === [16315.368812] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [16317.543979] Key type lgssc unregistered [16317.844758] LNet: 531732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16317.855138] LNetError: 531732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16318.885849] LNet: Removed LNI 192.168.202.114@tcp [16319.676258] Key type .llcrypt unregistered [16319.677883] Key type ._llcrypt unregistered