[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 402320798 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003228] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008895] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009012] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010005] pid_max: default: 32768 minimum: 301 [ 0.011090] LSM: Security Framework initializing [ 0.012025] Yama: becoming mindful. [ 0.013019] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.022745] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027532] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028122] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030085] *** VALIDATE tmpfs *** [ 0.032442] *** VALIDATE proc *** [ 0.033215] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035246] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038018] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.042296] debug: unmapping init [mem 0xfffffffface59000-0xfffffffface60fff] [ 0.044810] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045445] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046010] ... version: 2 [ 0.046795] ... bit width: 48 [ 0.047006] ... generic registers: 4 [ 0.047959] ... value mask: 0000ffffffffffff [ 0.048007] ... max period: 00007fffffffffff [ 0.049005] ... fixed-purpose events: 3 [ 0.050006] ... event mask: 000000070000000f [ 0.051258] rcu: Hierarchical SRCU implementation. [ 0.053211] smp: Bringing up secondary CPUs ... [ 0.054485] x86: Booting SMP configuration: [ 0.055019] .... node #0, CPUs: #1 #2 #3 [ 0.057818] smp: Brought up 1 node, 4 CPUs [ 0.058828] smpboot: Max logical packages: 1 [ 0.059006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.245132] node 0 deferred pages initialised in 186ms [ 0.248245] devtmpfs: initialized [ 0.249239] x86/mm: Memory block size: 128MB [ 0.251324] gcov: version magic: 0x41383552 [ 0.253281] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.254044] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.255227] pinctrl core: initialized pinctrl subsystem [ 0.256135] [ 0.256722] ************************************************************* [ 0.257011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258008] ** ** [ 0.259009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.260006] ** ** [ 0.261008] ** This means that this kernel is built to expose internal ** [ 0.262009] ** IOMMU data structures, which may compromise security on ** [ 0.263008] ** your system. ** [ 0.264006] ** ** [ 0.265008] ** If you see this message and you are not debugging the ** [ 0.266006] ** kernel, report this immediately to your vendor! ** [ 0.267008] ** ** [ 0.268009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269006] ************************************************************* [ 0.270582] NET: Registered protocol family 16 [ 0.271301] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.273031] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.274027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277032] cpuidle: using governor menu [ 0.277763] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.279372] PCI: Using configuration type 1 for base access [ 0.281115] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.290089] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.291017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.293101] cryptd: max_cpu_qlen set to 1000 [ 0.295149] ACPI: Added _OSI(Module Device) [ 0.296009] ACPI: Added _OSI(Processor Device) [ 0.296850] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.298022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.301682] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.306384] ACPI: Interpreter enabled [ 0.307035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.307848] ACPI: Using IOAPIC for interrupt routing [ 0.308053] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.310228] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.316901] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.318017] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.320008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.322031] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.324886] acpiphp: Slot [2] registered [ 0.325071] acpiphp: Slot [5] registered [ 0.326020] acpiphp: Slot [6] registered [ 0.326870] acpiphp: Slot [7] registered [ 0.328062] acpiphp: Slot [8] registered [ 0.328799] acpiphp: Slot [9] registered [ 0.330060] acpiphp: Slot [10] registered [ 0.330792] acpiphp: Slot [3] registered [ 0.331043] acpiphp: Slot [4] registered [ 0.331822] acpiphp: Slot [11] registered [ 0.333044] acpiphp: Slot [12] registered [ 0.333936] acpiphp: Slot [13] registered [ 0.334044] acpiphp: Slot [14] registered [ 0.334802] acpiphp: Slot [15] registered [ 0.336058] acpiphp: Slot [16] registered [ 0.337013] acpiphp: Slot [17] registered [ 0.337789] acpiphp: Slot [18] registered [ 0.338184] acpiphp: Slot [19] registered [ 0.340065] acpiphp: Slot [20] registered [ 0.341072] acpiphp: Slot [21] registered [ 0.343074] acpiphp: Slot [22] registered [ 0.345053] acpiphp: Slot [23] registered [ 0.346082] acpiphp: Slot [24] registered [ 0.347113] acpiphp: Slot [25] registered [ 0.349075] acpiphp: Slot [26] registered [ 0.351052] acpiphp: Slot [27] registered [ 0.351968] acpiphp: Slot [28] registered [ 0.352046] acpiphp: Slot [29] registered [ 0.352998] acpiphp: Slot [30] registered [ 0.355071] acpiphp: Slot [31] registered [ 0.356053] PCI host bridge to bus 0000:00 [ 0.357014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.360014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.362012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.364009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.365008] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.367011] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.368125] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.369712] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.371814] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.382066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388312] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.390576] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.392020] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.393532] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397009] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.410008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414099] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431011] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441024] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.452013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.471011] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.475014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.479009] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.489011] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.496906] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.500008] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.505009] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.515010] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.521111] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.526010] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.529008] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.538010] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.547156] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.551011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.554008] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.564011] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.571888] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.573207] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.575208] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.577205] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.578109] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.581194] iommu: Default domain type: Passthrough [ 0.582364] SCSI subsystem initialized [ 0.583069] ACPI: bus type USB registered [ 0.584060] usbcore: registered new interface driver usbfs [ 0.586030] usbcore: registered new interface driver hub [ 0.587031] usbcore: registered new device driver usb [ 0.588126] pps_core: LinuxPPS API ver. 1 registered [ 0.589008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.591032] PTP clock support registered [ 0.592080] EDAC MC: Ver: 3.0.0 [ 0.594109] PCI: Using ACPI for IRQ routing [ 0.594931] NetLabel: Initializing [ 0.596009] NetLabel: domain hash size = 128 [ 0.596965] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.598108] NetLabel: unlabeled traffic allowed by default [ 0.600081] vgaarb: loaded [ 0.600673] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.602006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.609286] clocksource: Switched to clocksource kvm-clock [ 0.693644] VFS: Disk quotas dquot_6.6.0 [ 0.694569] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.695958] *** VALIDATE ramfs *** [ 0.696561] *** VALIDATE hugetlbfs *** [ 0.697401] pnp: PnP ACPI init [ 0.698959] pnp: PnP ACPI: found 6 devices [ 0.712914] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.714809] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.715971] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.717343] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.718797] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.720148] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.721809] NET: Registered protocol family 2 [ 0.723465] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.728101] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.733194] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.740416] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.744353] TCP: Hash tables configured (established 65536 bind 65536) [ 0.746617] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.750889] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.754052] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.756749] NET: Registered protocol family 1 [ 0.758973] RPC: Registered named UNIX socket transport module. [ 0.760506] RPC: Registered udp transport module. [ 0.761736] RPC: Registered tcp transport module. [ 0.764030] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.766658] NET: Registered protocol family 44 [ 0.767921] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.769311] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.770656] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.772305] PCI: CLS 0 bytes, default 64 [ 0.773947] Unpacking initramfs... [ 1.990722] debug: unmapping init [mem 0xffff8fb07cc54000-0xffff8fb07ffbffff] [ 1.993621] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.995223] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.997174] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.453972] Initialise system trusted keyrings [ 2.455882] Key type blacklist registered [ 2.457100] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.463438] zbud: loaded [ 2.465613] *** VALIDATE nfs *** [ 2.466476] *** VALIDATE nfs4 *** [ 2.467944] pstore: using deflate compression [ 2.469835] Platform Keyring initialized [ 2.535792] NET: Registered protocol family 38 [ 2.536949] Key type asymmetric registered [ 2.537907] Asymmetric key parser 'x509' registered [ 2.539072] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.540858] io scheduler mq-deadline registered [ 2.541865] io scheduler kyber registered [ 2.542883] io scheduler bfq registered [ 2.543964] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.545713] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.548273] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.551830] ACPI: Power Button [PWRF] [ 2.557890] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.564882] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.576402] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.581701] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.590629] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.616540] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.642422] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.647144] Non-volatile memory driver v1.3 [ 2.648437] Linux agpgart interface v0.103 [ 2.674576] virtio_blk virtio1: [vda] 135560 512-byte logical blocks (69.4 MB/66.2 MiB) [ 2.677776] vda: detected capacity change from 0 to 69406720 [ 2.689485] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.691520] vdb: detected capacity change from 0 to 1073741824 [ 2.702818] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.705312] vdc: detected capacity change from 0 to 2621440000 [ 2.716349] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.718490] vdd: detected capacity change from 0 to 2621440000 [ 2.729285] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.731464] vde: detected capacity change from 0 to 4294967296 [ 2.742338] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.744156] vdf: detected capacity change from 0 to 4294967296 [ 2.748585] libphy: Fixed MDIO Bus: probed [ 2.752039] usbcore: registered new interface driver usbserial_generic [ 2.753749] usbserial: USB Serial support registered for generic [ 2.755110] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.758389] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.759481] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.761239] mousedev: PS/2 mouse device common for all mice [ 2.763212] rtc_cmos 00:05: RTC can wake from S4 [ 2.763731] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.766134] rtc_cmos 00:05: registered as rtc0 [ 2.769215] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.769559] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.772358] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.774383] intel_pstate: CPU model not supported [ 2.779017] hid: raw HID events driver (C) Jiri Kosina [ 2.780902] usbcore: registered new interface driver usbhid [ 2.783068] usbhid: USB HID core driver [ 2.784538] drop_monitor: Initializing network drop monitor service [ 2.786397] Initializing XFRM netlink socket [ 2.788455] NET: Registered protocol family 10 [ 2.790856] Segment Routing with IPv6 [ 2.792087] NET: Registered protocol family 17 [ 2.793367] mpls_gso: MPLS GSO support [ 2.797704] RAS: Correctable Errors collector initialized. [ 2.799008] AVX version of gcm_enc/dec engaged. [ 2.799845] AES CTR mode by8 optimization enabled [ 2.849127] sched_clock: Marking stable (2849102603, 0)->(3516686646, -667584043) [ 2.861022] registered taskstats version 1 [ 2.862329] Loading compiled-in X.509 certificates [ 2.863494] zswap: loaded using pool lzo/zbud [ 2.893137] Key type big_key registered [ 2.901339] Key type encrypted registered [ 2.902288] ima: No TPM chip found, activating TPM-bypass! [ 2.903362] ima: Allocated hash algorithm: sha1 [ 2.904254] ima: No architecture policies found [ 2.905203] evm: Initialising EVM extended attributes: [ 2.906118] evm: security.selinux [ 2.906775] evm: security.ima [ 2.907424] evm: security.capability [ 2.908093] evm: HMAC attrs: 0x1 [ 2.909371] rtc_cmos 00:05: setting system clock to 2026-03-19 17:31:36 UTC (1773941496) [ 2.914147] debug: unmapping init [mem 0xffffffffade03000-0xffffffffadffffff] [ 2.915635] debug: unmapping init [mem 0xffffffffacb82000-0xfffffffface58fff] [ 2.923095] Write protecting the kernel read-only data: 28672k [ 2.925042] debug: unmapping init [mem 0xffffffffab203000-0xffffffffab3fffff] [ 2.926657] debug: unmapping init [mem 0xffffffffabb14000-0xffffffffabbfffff] [ 2.956295] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.961404] systemd[1]: Detected virtualization kvm. [ 2.962613] systemd[1]: Detected architecture x86-64. [ 2.963634] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.983175] systemd[1]: No hostname configured. [ 2.984156] systemd[1]: Set hostname to . [ 2.985320] random: systemd: uninitialized urandom read (16 bytes read) [ 2.986614] systemd[1]: Initializing machine ID from random generator. [ 3.028786] random: ln: uninitialized urandom read (6 bytes read) [ 3.083374] random: systemd: uninitialized urandom read (16 bytes read) [ 3.085562] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.089249] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.092352] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.503584] device-mapper: uevent: version 1.0.3 [ 3.505213] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 3.758043] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.037268] virtio_net virtio0 ens2: renamed from eth0 [ 4.051816] scsi host0: ata_piix [ 4.071275] scsi host1: ata_piix [ 4.078412] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.080848] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.299750] dracut-initqueue[583]: RTNETLINK answers: File exists [ 8.411490] random: crng init done [ 8.412459] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 8.832594] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.785220] printk: systemd: 26 output lines suppressed due to ratelimiting [ 9.980758] SELinux: Disabled at runtime. [ 10.027410] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.032876] systemd[1]: Detected virtualization kvm. [ 10.033944] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.421462] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.424889] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.428655] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.432207] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.436250] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.443174] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.452640] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ 10.530337] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 10.907318] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 11.228769] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.291506] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.400907] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.411180] EDAC sbridge: Ver: 1.1.2 [ 12.761515] Key type dns_resolver registered [ 13.056152] NFS: Registering the id_resolver key type [ 13.057613] Key type id_resolver registered [ 13.058621] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 34.918091] libcfs: loading out-of-tree module taints kernel. [ 34.937498] Key type ._llcrypt registered [ 34.938882] Key type .llcrypt registered [ 34.978856] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 42.184339] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 42.901412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 42.908452] alg: No test for adler32 (adler32-zlib) [ 43.894821] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 44.188426] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 45.799194] Key type lgssc registered [ 46.342845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 52.328397] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.634504] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 70.270112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 70.279243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 71.393015] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 71.406538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 71.448644] Lustre: lustre-MDT0000: new disk, initializing [ 71.475466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 71.483515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 72.821849] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 77.998061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 78.034237] Lustre: 6468:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 78.045745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 78.048678] Lustre: Skipped 1 previous similar message [ 78.091435] Lustre: lustre-MDT0001: new disk, initializing [ 78.115486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 78.125223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 78.129578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 79.389274] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 81.710337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 85.113115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 85.244827] Lustre: lustre-OST0000: new disk, initializing [ 85.246726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 85.262886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.630291] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.115728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 88.120838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 88.146615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 94.049425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 94.106454] Lustre: lustre-OST0001: new disk, initializing [ 94.108487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 94.127220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 96.097343] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.692073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 96.695947] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 96.720179] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 102.522968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 105.153483] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 112.403536] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 113.861177] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 115.370549] Lustre: DEBUG MARKER: Client: 2.17.50.193 [ 116.195049] Lustre: DEBUG MARKER: MDS: 2.17.50.193 [ 117.059536] Lustre: DEBUG MARKER: OSS: 2.17.50.193 [ 117.647420] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Mar 19 13:33:29 EDT 2026 [ 124.350442] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 124.896315] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 137.695883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.696483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.701041] Lustre: Skipped 2 previous similar messages [ 137.703586] Lustre: Skipped 1 previous similar message [ 142.816483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 142.820397] Lustre: Skipped 5 previous similar messages [ 143.073718] Lustre: server umount lustre-MDT0000 complete [ 144.499259] LustreError: 10230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773941638 with bad export cookie 12289857724905801122 [ 144.501343] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 144.506143] LustreError: 10230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 147.936588] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 147.937112] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 147.944238] Lustre: Skipped 2 previous similar messages [ 149.983834] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 149.985950] Lustre: Skipped 2 previous similar messages [ 150.733665] Lustre: server umount lustre-MDT0001 complete [ 158.550482] Lustre: server umount lustre-OST0000 complete [ 165.970770] Lustre: server umount lustre-OST0001 complete [ 168.172975] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 171.244652] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 186.794644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.950184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.963076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.010672] Lustre: lustre-MDT0000: new disk, initializing [ 187.038438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.046716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.657532] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 193.231087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.282149] Lustre: 14936:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 193.422155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 194.994357] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 198.862044] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 200.130344] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 202.476890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.611996] Lustre: lustre-OST0000: new disk, initializing [ 202.613701] Lustre: Skipped 1 previous similar message [ 202.615933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.617395] Lustre: Skipped 2 previous similar messages [ 202.652554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.655588] Lustre: Skipped 1 previous similar message [ 204.438271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.444402] Lustre: Skipped 1 previous similar message [ 204.449438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.501989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.286253] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 209.274532] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 210.746123] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 210.820366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 212.152123] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 212.231486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 215.009960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 215.010096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.011739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 215.011922] LustreError: Skipped 1 previous similar message [ 215.014791] Lustre: Skipped 1 previous similar message [ 225.252241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 225.255692] Lustre: Skipped 4 previous similar messages [ 227.295096] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 227.347442] Lustre: server umount lustre-OST0000 complete [ 229.344410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 229.348093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 243.167103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 243.240058] Lustre: server umount lustre-MDT0000 complete [ 244.541583] LustreError: 16679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773941738 with bad export cookie 12289857724905802634 [ 244.547901] LustreError: 16679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 244.549900] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 244.691329] Lustre: server umount lustre-MDT0001 complete [ 246.372224] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:35:38 (1773941738) [ 247.337224] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 247.956188] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:35:40 (1773941740) [ 251.003978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 251.182453] LustreError: 18940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 251.189382] LustreError: 18940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 251.205691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.422864] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 255.834430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.320187] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 259.345557] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 260.861956] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 263.952289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.861881] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 269.338866] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 286.695254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 286.696245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.700507] LustreError: Skipped 1 previous similar message [ 286.709184] Lustre: Skipped 1 previous similar message [ 286.710874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 286.721850] Lustre: Skipped 8 previous similar messages [ 289.585492] Lustre: server umount lustre-OST0000 complete [ 291.810214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 297.646923] Lustre: server umount lustre-MDT0000 complete [ 299.244951] LustreError: 18923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773941792 with bad export cookie 12289857724905803523 [ 299.250762] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.392161] Lustre: server umount lustre-MDT0001 complete [ 303.946896] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 305.382589] Key type lgssc unregistered [ 305.576554] LNet: 22138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.581543] LNetError: 22138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 305.592740] LNet: Removed LNI 192.168.202.116@tcp [ 306.127194] Key type .llcrypt unregistered [ 306.129191] Key type ._llcrypt unregistered [ 316.795993] Key type ._llcrypt registered [ 316.797446] Key type .llcrypt registered [ 316.856898] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 317.231744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 317.251793] alg: No test for adler32 (adler32-zlib) [ 318.103450] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 318.196124] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 319.783226] Key type lgssc registered [ 320.391981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 324.023651] loop: module loaded [ 324.059248] loop0: detected capacity change from 0 to 204800000 [ 324.189112] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 324.192660] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 324.362672] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 327.212479] loop0: detected capacity change from 0 to 204800000 [ 327.316052] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 327.321473] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 327.327430] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 329.700946] loop0: detected capacity change from 0 to 204800000 [ 332.227415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 332.235612] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 333.364199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 333.383482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 333.431830] Lustre: lustre-MDT0000: new disk, initializing [ 333.458723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.468537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 335.118107] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 339.775592] loop1: detected capacity change from 0 to 204800000 [ 342.400142] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 342.557884] Lustre: lustre-OST07c6: new disk, initializing [ 342.562297] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 342.566969] Lustre: Skipped 1 previous similar message [ 342.612828] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 345.105641] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 350.224085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 350.229855] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 350.245355] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 352.836739] Lustre: 24281:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 352.841352] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 352.846068] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 352.851628] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 352.857700] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 352.863358] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 369.402082] Lustre: 24281:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 369.406232] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 369.409771] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 369.414266] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 369.419206] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 369.423518] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 373.225362] Lustre: 24281:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 373.231244] Lustre: 24281:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 373.236084] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 373.240675] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 373.245756] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 373.251024] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 373.256695] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 373.262325] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 373.267791] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 373.273034] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 373.278225] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 373.282647] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 376.356326] Lustre: 24280:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 376.362335] Lustre: 24280:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 376.367152] Lustre: 24280:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 376.372023] Lustre: 24280:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 376.377118] Lustre: 24280:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 376.382119] Lustre: 24280:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 376.387029] Lustre: 24280:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 376.392279] Lustre: 24280:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 376.397426] Lustre: 24280:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 376.402118] Lustre: 24280:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 376.406928] Lustre: 24280:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 376.411563] Lustre: 24280:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.229858] Lustre: 24282:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 383.238921] Lustre: 24282:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 383.246202] Lustre: 24282:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 383.253650] Lustre: 24282:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 383.260855] Lustre: 24282:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 383.268198] Lustre: 24282:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 383.275428] Lustre: 24282:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 383.283155] Lustre: 24282:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 383.290494] Lustre: 24282:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 383.297117] Lustre: 24282:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 383.303960] Lustre: 24282:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 383.309845] Lustre: 24282:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 393.795214] Lustre: 24281:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 393.801869] Lustre: 24281:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 393.807140] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 393.811990] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 393.817441] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 393.822946] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 393.828303] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 393.834051] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 393.839441] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 393.844508] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 393.849865] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 393.855126] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 437.217974] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.218647] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.225603] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 441.445616] Lustre: server umount lustre-OST07c6 complete [ 445.822392] Lustre: server umount lustre-MDT0000 complete [ 452.742375] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 454.149782] Key type lgssc unregistered [ 454.316691] LNet: 27800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.323831] LNetError: 27800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.336668] LNet: Removed LNI 192.168.202.116@tcp [ 454.812132] Key type .llcrypt unregistered [ 454.814273] Key type ._llcrypt unregistered [ 462.636392] Key type ._llcrypt registered [ 462.637672] Key type .llcrypt registered [ 462.693859] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 463.080184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 463.091580] alg: No test for adler32 (adler32-zlib) [ 463.954214] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 464.037035] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 465.623145] Key type lgssc registered [ 466.141622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 469.265323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 469.274798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.477301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.135715] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 475.381432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.613204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 477.009231] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 478.892442] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 480.014317] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 482.638341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.805073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.620199] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 487.071493] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 496.116661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 505.828711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 505.829066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.832163] LustreError: Skipped 1 previous similar message [ 505.837974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.341662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.343178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.349611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.287910] Lustre: server umount lustre-OST0000 complete [ 511.458138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 511.458299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.459380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 511.459388] Lustre: Skipped 1 previous similar message [ 511.477995] Lustre: Skipped 1 previous similar message [ 516.577315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.580382] Lustre: Skipped 2 previous similar messages [ 517.298058] Lustre: server umount lustre-MDT0000 complete [ 518.802586] LustreError: 29018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942012 with bad export cookie 4031222865174642175 [ 518.806639] LustreError: 29018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 518.807600] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.957565] Lustre: server umount lustre-MDT0001 complete [ 523.470099] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 524.937101] Key type lgssc unregistered [ 525.112561] LNet: 32436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 525.116731] LNetError: 32436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 525.127796] LNet: Removed LNI 192.168.202.116@tcp [ 525.601647] Key type .llcrypt unregistered [ 525.604307] Key type ._llcrypt unregistered [ 533.405591] Key type ._llcrypt registered [ 533.407097] Key type .llcrypt registered [ 533.447948] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 533.807510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 533.814247] alg: No test for adler32 (adler32-zlib) [ 534.667936] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 534.750933] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 536.335136] Key type lgssc registered [ 536.838902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 539.784017] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 539.791610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.967380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.306552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 545.566283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.688171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 547.182143] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 549.275079] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.525135] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 553.029334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 553.146222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.428810] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 557.233766] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 566.446730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 571.878175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 571.879332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.880637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 571.884198] LustreError: Skipped 1 previous similar message [ 571.892185] Lustre: Skipped 1 previous similar message [ 576.993852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 576.996587] Lustre: Skipped 2 previous similar messages [ 577.684772] Lustre: server umount lustre-OST0000 complete [ 581.599925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 581.602175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.606923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 587.233086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 587.234840] Lustre: Skipped 3 previous similar messages [ 592.352629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 592.354758] Lustre: Skipped 1 previous similar message [ 593.375173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 593.472460] Lustre: server umount lustre-MDT0000 complete [ 594.826602] LustreError: 33645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942088 with bad export cookie 44156759791266030 [ 594.833289] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.986662] Lustre: server umount lustre-MDT0001 complete [ 598.750645] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 599.792884] Key type lgssc unregistered [ 599.925507] LNet: 36865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.928697] LNetError: 36865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.939394] LNet: Removed LNI 192.168.202.116@tcp [ 600.343682] Key type .llcrypt unregistered [ 600.345822] Key type ._llcrypt unregistered [ 609.375465] Key type ._llcrypt registered [ 609.377035] Key type .llcrypt registered [ 609.416296] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 609.772987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 609.786078] alg: No test for adler32 (adler32-zlib) [ 610.641480] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 610.742395] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 612.343111] Key type lgssc registered [ 612.740269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 616.088655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 616.100740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.311811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.039874] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 622.380372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.519446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 624.250722] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 626.743785] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 627.789207] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 630.655700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 630.876964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 632.862934] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 634.697994] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 642.150630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 647.648661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.650483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.652132] Lustre: Skipped 2 previous similar messages [ 647.654210] Lustre: Skipped 1 previous similar message [ 650.218217] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 650.222644] Lustre: Skipped 1 previous similar message [ 652.769857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.773962] Lustre: Skipped 2 previous similar messages [ 653.688456] Lustre: server umount lustre-MDT0000 complete [ 655.275075] LustreError: 38220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942148 with bad export cookie 8254466979539956843 [ 655.280685] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.281501] LustreError: 38220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 655.331104] LustreError: 38235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.338190] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 655.340204] LustreError: 38235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 657.888878] LustreError: 38237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.889364] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.451875] LustreError: 40389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.452952] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 660.469332] Lustre: Skipped 1 previous similar message [ 661.521277] Lustre: server umount lustre-MDT0001 complete [ 680.041573] Lustre: server umount lustre-OST0000 complete [ 687.424716] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 688.645757] Key type lgssc unregistered [ 688.765346] LNet: 41767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.767475] LNetError: 41767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.773428] LNet: Removed LNI 192.168.202.116@tcp [ 689.071985] Key type .llcrypt unregistered [ 689.073090] Key type ._llcrypt unregistered [ 699.998395] Key type ._llcrypt registered [ 699.999871] Key type .llcrypt registered [ 700.063879] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 700.423921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.463070] alg: No test for adler32 (adler32-zlib) [ 701.323095] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 701.411976] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 702.999201] Key type lgssc registered [ 703.619989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 707.154072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 707.165551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.373475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.782442] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 713.795880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.952624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.580295] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 718.118068] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 719.675759] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 723.058945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 723.254518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.668995] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 728.235956] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 732.462946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 737.762799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 737.766789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.768201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 737.780523] Lustre: Skipped 1 previous similar message [ 738.803145] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 738.807703] Lustre: Skipped 1 previous similar message [ 742.321237] Lustre: server umount lustre-OST0000 complete [ 755.609362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.615700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.665818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.665998] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.674120] Lustre: Skipped 1 previous similar message [ 761.885824] Lustre: server umount lustre-MDT0000 complete [ 762.848660] LustreError: 43144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.855104] LustreError: 43144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 763.480358] LustreError: 43169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942257 with bad export cookie 2558733999805108288 [ 763.484980] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.645775] Lustre: server umount lustre-MDT0001 complete [ 772.978484] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 773.950899] Key type lgssc unregistered [ 774.068333] LNet: 46500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.071030] LNetError: 46500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.079289] LNet: Removed LNI 192.168.202.116@tcp [ 774.326078] Key type .llcrypt unregistered [ 774.327103] Key type ._llcrypt unregistered [ 784.096531] Key type ._llcrypt registered [ 784.098363] Key type .llcrypt registered [ 784.161447] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 784.496801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 784.539652] alg: No test for adler32 (adler32-zlib) [ 785.414439] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 785.503646] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 787.111194] Key type lgssc registered [ 787.562718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 791.061873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 791.074141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.332213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.109915] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 797.899077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.084501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 799.595874] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 801.336288] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 802.418701] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 804.562872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.685320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.322436] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 808.107983] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 817.005586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 821.222296] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 822.888562] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 833.661828] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-1913h45m25s [ 834.163850] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 837.602724] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.603991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 837.605758] Lustre: Skipped 2 previous similar messages [ 837.607211] Lustre: Skipped 1 previous similar message [ 838.627469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.629058] Lustre: Skipped 1 previous similar message [ 840.162864] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 840.166051] Lustre: Skipped 1 previous similar message [ 843.746722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 849.375150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 849.439397] Lustre: server umount lustre-MDT0000 complete [ 850.567587] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 850.916779] LustreError: 47876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.923831] LustreError: 47876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 851.426632] LustreError: 47901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.859187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.903118] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 853.002634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.021930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 854.056215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 854.325946] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 855.411476] Lustre: DEBUG MARKER: Wait for df (35962) ... [ 855.921666] Lustre: DEBUG MARKER: done [ 856.243444] LustreError: 47855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.16@tcp arrived at 1773942349 with bad export cookie 14001758309568886380 [ 856.247355] LustreError: 47855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 858.080569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.082855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.086848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.089084] Lustre: Skipped 4 previous similar messages [ 859.108243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 859.113535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.016538] Lustre: server umount lustre-OST0000 complete [ 869.346139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 869.346303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 869.347175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 869.347182] Lustre: Skipped 2 previous similar messages [ 869.355481] Lustre: Skipped 1 previous similar message [ 871.024620] Lustre: server umount lustre-MDT0000 complete [ 872.775493] LustreError: 50187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942366 with bad export cookie 14001758309568886849 [ 872.781370] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 872.786154] LustreError: 50187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 872.928783] Lustre: server umount lustre-MDT0001 complete [ 876.956796] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 877.853950] Key type lgssc unregistered [ 877.975338] LNet: 52446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.977695] LNetError: 52446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.985302] LNet: Removed LNI 192.168.202.116@tcp [ 878.268386] Key type .llcrypt unregistered [ 878.269869] Key type ._llcrypt unregistered [ 893.417897] Key type ._llcrypt registered [ 893.419014] Key type .llcrypt registered [ 893.455559] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 893.773295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.808062] alg: No test for adler32 (adler32-zlib) [ 894.651683] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 894.729319] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 896.311163] Key type lgssc registered [ 896.668133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 899.395099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 899.401486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.516979] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 900.526234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 900.622571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.052790] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 905.051322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.082038] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 905.092488] Lustre: 54614:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 905.197423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 906.521605] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 908.266396] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 909.391974] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 911.491238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 911.566503] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 911.625516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.105517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 913.301453] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 914.995512] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 918.285473] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-1913h46m50s [ 918.765155] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 920.939541] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 921.727369] Lustre: DEBUG MARKER: Creating a test file and stat it [ 921.742592] LustreError: 56337:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 942.060517] Lustre: lustre-MDT0000: Client 64826859-e9cb-4dc3-a418-fb84343f2e21 (at 192.168.202.16@tcp) reconnecting [ 961.783052] LustreError: 56337:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 962.746196] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 964.005805] Lustre: DEBUG MARKER: done [ 968.162777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 968.166946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.174290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.675827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.681057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.690398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 971.931540] Lustre: server umount lustre-OST0000 complete [ 973.793234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.794242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 973.796320] Lustre: Skipped 1 previous similar message [ 979.688049] Lustre: server umount lustre-MDT0000 complete [ 980.934607] LustreError: 54605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942474 with bad export cookie 7814044147407721664 [ 980.938981] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.940884] LustreError: 54605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 981.078766] Lustre: server umount lustre-MDT0001 complete [ 984.996744] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 986.341338] Key type lgssc unregistered [ 986.510707] LNet: 58579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.515514] LNetError: 58579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.525553] LNet: Removed LNI 192.168.202.116@tcp [ 987.006214] Key type .llcrypt unregistered [ 987.008917] Key type ._llcrypt unregistered [ 1006.305187] Key type ._llcrypt registered [ 1006.306623] Key type .llcrypt registered [ 1006.357685] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1006.792377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1006.819074] alg: No test for adler32 (adler32-zlib) [ 1007.744293] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1007.847533] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1009.463174] Key type lgssc registered [ 1010.137516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1011.772439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1011.779740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.909611] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1012.924992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1013.018808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.498176] Lustre: Failing over lustre-MDT0000 [ 1013.654873] Lustre: server umount lustre-MDT0000 complete [ 1016.806833] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:48:28 (1773942508) [ 1020.052930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.312844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.843301] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1024.704234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.751968] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1024.763378] Lustre: 61341:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1024.904528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.407255] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1029.224425] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1030.747979] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1033.742344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.835596] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1033.931628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.495557] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1039.161364] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1041.435260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1048.677881] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1051.620211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.622766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.628044] Lustre: Skipped 2 previous similar messages [ 1051.634672] Lustre: Skipped 2 previous similar messages [ 1056.737540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.740644] Lustre: Skipped 2 previous similar messages [ 1057.256084] Lustre: server umount lustre-MDT0000 complete [ 1058.942601] LustreError: 61333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942552 with bad export cookie 5978483679924209417 [ 1058.949631] LustreError: 61333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1058.953717] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.134226] Lustre: server umount lustre-MDT0001 complete [ 1060.659082] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1062.944792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1066.941607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.023263] LustreError: 64753:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1069.023875] LustreError: 60263:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a64e3179 x1860113531961984/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1069.280798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.336143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1070.988708] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1074.659984] LustreError: 62962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.661248] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.665038] LustreError: 62962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1075.052392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.007727] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1078.304105] Lustre: 60266:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773942555/real 1773942555] req@000000005df8b5e1 x1860113531961600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773942571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1079.293775] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1080.294057] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1080.815411] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1081.311202] Lustre: 60267:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773942557/real 1773942557] req@000000003f85ac5a x1860113531961728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773942573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1082.342212] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1084.191371] Lustre: 60265:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773942560/real 1773942560] req@00000000515344ff x1860113531961856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773942576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1085.409841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1085.414248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.421141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.460137] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.464150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.467774] Lustre: Skipped 1 previous similar message [ 1087.473817] Lustre: Skipped 1 previous similar message [ 1089.409253] Lustre: server umount lustre-MDT0000 complete [ 1091.274070] LustreError: 62058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942584 with bad export cookie 5978483679924211685 [ 1091.279753] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1091.280972] LustreError: 62058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1091.428612] Lustre: server umount lustre-MDT0001 complete [ 1094.690410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1098.665828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.704598] LustreError: 60263:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b5bd93a9 x1860113531982464/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1100.720596] LustreError: 60263:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 1100.869760] LustreError: 65985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.903539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.907418] Lustre: Skipped 1 previous similar message [ 1100.933388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1102.741218] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1106.403933] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.404390] LustreError: 65483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.414856] LustreError: 65483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1106.658884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.881585] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1108.442911] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1108.896193] Lustre: 60267:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773942586/real 1773942586] req@00000000d43ffdd3 x1860113531982080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773942602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1108.911321] Lustre: 60267:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1110.716605] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1112.335233] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1112.991149] Lustre: 60267:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773942590/real 1773942590] req@000000000c001c03 x1860113531982208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773942606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1117.157908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1117.162441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.170322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1117.175622] Lustre: Skipped 1 previous similar message [ 1120.867631] Lustre: server umount lustre-MDT0000 complete [ 1122.265703] LustreError: 61331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942615 with bad export cookie 5978483679924213064 [ 1122.270371] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.274485] LustreError: 61331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1122.388366] Lustre: server umount lustre-MDT0001 complete [ 1125.331388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1126.982253] Lustre: DEBUG MARKER: files compared the same [ 1138.384643] Lustre: server umount lustre-OST0000 complete [ 1145.595633] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1147.279442] Key type lgssc unregistered [ 1147.467767] LNet: 70256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.472872] LNetError: 70256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.482853] LNet: Removed LNI 192.168.202.116@tcp [ 1148.007095] Key type .llcrypt unregistered [ 1148.010161] Key type ._llcrypt unregistered [ 1158.849195] Key type ._llcrypt registered [ 1158.851444] Key type .llcrypt registered [ 1158.905377] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1159.393192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.406111] alg: No test for adler32 (adler32-zlib) [ 1160.278729] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1160.377880] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1161.975306] Key type lgssc registered [ 1162.614292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1166.868606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1166.880573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.072838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.947247] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug +malloc all [ 1173.916646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.047425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1175.454406] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug +malloc all [ 1177.843519] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1179.803680] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1182.877294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1182.989306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.547886] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug +malloc all [ 1187.801513] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1193.250916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1203.681491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.681540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1203.683105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1203.689533] Lustre: Skipped 1 previous similar message [ 1206.810196] Lustre: server umount lustre-OST0000 complete [ 1208.800225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.800819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.811192] Lustre: Skipped 1 previous similar message [ 1208.814910] Lustre: Skipped 1 previous similar message [ 1213.919596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.921780] Lustre: Skipped 2 previous similar messages [ 1214.416102] Lustre: server umount lustre-MDT0000 complete [ 1216.132368] LustreError: 71615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942709 with bad export cookie 8334243872557288691 [ 1216.134939] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1216.140455] LustreError: 71615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1216.295103] Lustre: server umount lustre-MDT0001 complete [ 1220.614590] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1222.189398] Key type lgssc unregistered [ 1222.363672] LNet: 74837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.368488] LNetError: 74837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.378701] LNet: Removed LNI 192.168.202.116@tcp [ 1222.868904] Key type .llcrypt unregistered [ 1222.871480] Key type ._llcrypt unregistered [ 1233.794275] Key type ._llcrypt registered [ 1233.795571] Key type .llcrypt registered [ 1233.835097] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1234.198797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1234.252445] alg: No test for adler32 (adler32-zlib) [ 1235.164779] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1235.288753] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1236.903158] Key type lgssc registered [ 1237.568681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.993732] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1241.002612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1256.479381] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1261.599211] LustreError: 76186:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 1288.223175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.381305] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1292.677510] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1295.729179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.766132] Lustre: *** cfs_fail_loc=706, val=0*** [ 1295.897122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.922269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1297.145384] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1300.524681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.642202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1301.938336] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1304.520953] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1305.776399] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1308.133893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.136310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.142246] Lustre: Skipped 1 previous similar message [ 1312.671487] Lustre: server umount lustre-OST0000 complete [ 1316.320154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1316.323403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.328193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1316.330049] Lustre: Skipped 1 previous similar message [ 1318.369246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1318.370321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.597588] Lustre: server umount lustre-MDT0000 complete [ 1321.729072] LustreError: 77869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773942815 with bad export cookie 16706557000661727369 [ 1321.732223] LustreError: 77869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1321.732920] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.838187] Lustre: server umount lustre-MDT0001 complete [ 1325.871819] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1326.946136] Key type lgssc unregistered [ 1327.077393] LNet: 79447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.080330] LNetError: 79447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.088499] LNet: Removed LNI 192.168.202.116@tcp [ 1327.493486] Key type .llcrypt unregistered [ 1327.495817] Key type ._llcrypt unregistered [ 1337.431653] Key type ._llcrypt registered [ 1337.432934] Key type .llcrypt registered [ 1337.483219] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1337.834631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1337.845314] alg: No test for adler32 (adler32-zlib) [ 1338.726602] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1338.831146] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1340.431200] Key type lgssc registered [ 1340.976280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1343.971342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1343.983289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.520604] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1349.832121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.065453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1351.616798] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1355.233961] LustreError: 81491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.243316] LustreError: 81491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1355.455902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1355.655562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.780747] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1359.778347] LustreError: 81495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.791674] LustreError: 81495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1361.574577] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.578666] Lustre: MGS: Client 0c870afe-8bbf-40a9-b08d-5d90235d886b (at 0@lo) reconnecting [ 1361.588777] LustreError: 81505:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1361.675838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.713479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1362.958131] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1372.130942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.134203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1372.134392] Lustre: Skipped 1 previous similar message [ 1372.135409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1377.252966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1377.254537] Lustre: Skipped 2 previous similar messages [ 1377.563143] Lustre: server umount lustre-OST0000 complete [ 1382.370943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.370984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.380122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.381868] Lustre: Skipped 1 previous similar message [ 1385.225331] Lustre: server umount lustre-MDT0000 complete [ 1386.768322] Lustre: server umount lustre-MDT0001 complete [ 1387.954321] Lustre: server umount MGS complete [ 1392.472610] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:54:44 (1773942884) [ 1403.346657] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 1407.138651] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1421.689679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.397855] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1427.777961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.862540] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1427.877317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1427.917941] Lustre: lustre-MDT0001: new disk, initializing [ 1427.946058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1427.956408] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1427.959038] Lustre: 88555:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1427.961506] LustreError: 88555:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1429.209562] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1434.318709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.459562] Lustre: lustre-OST0000: new disk, initializing [ 1434.462711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1434.498934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.127837] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1439.200084] Lustre: 89742:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000060cf5005 x1860113879041920/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773942948 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1440.941588] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1443.736792] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1443.742201] Lustre: MGS: Client abb0be45-e368-4a34-a466-590e4f848ad8 (at 0@lo) reconnecting [ 1443.767692] Lustre: 90233:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1443.774762] LustreError: 88540:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1443.789592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1443.830686] Lustre: lustre-MDT0000: new disk, initializing [ 1443.899917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.912891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1444.960648] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1444.963758] Lustre: 90260:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000927396da x1860113879047680/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1773942954 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1445.649638] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1447.012311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1447.017273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1447.086932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1467.363730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.364961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1467.371809] Lustre: Skipped 2 previous similar messages [ 1467.377980] Lustre: Skipped 1 previous similar message [ 1472.481140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1472.483777] Lustre: Skipped 1 previous similar message [ 1480.159117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1480.216490] Lustre: server umount lustre-OST0000 complete [ 1482.720894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1482.721254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.725027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.728770] Lustre: Skipped 1 previous similar message [ 1482.730677] Lustre: Skipped 2 previous similar messages [ 1487.887238] Lustre: server umount lustre-MDT0000 complete [ 1495.351612] Lustre: server umount lustre-MDT0001 complete [ 1496.715472] Lustre: server umount MGS complete [ 1500.726055] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:56:32 (1773942992) [ 1506.895649] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1507.934073] Key type lgssc unregistered [ 1508.066510] LNet: 92722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.068787] LNetError: 92722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.077620] LNet: Removed LNI 192.168.202.116@tcp [ 1508.398132] Key type .llcrypt unregistered [ 1508.399108] Key type ._llcrypt unregistered [ 1515.654177] Key type ._llcrypt registered [ 1515.655358] Key type .llcrypt registered [ 1515.694797] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1516.030573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1516.041271] alg: No test for adler32 (adler32-zlib) [ 1516.899876] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1516.987713] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1518.567197] Key type lgssc registered [ 1518.940322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1521.544330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1521.550487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1522.709333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.954122] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1526.926618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.055900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1528.260339] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1529.962113] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1531.150097] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1533.443919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1533.560391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.342918] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1536.973314] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1547.875781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1557.472612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1557.475630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.481169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1558.497970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1558.498228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1558.498592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.499391] Lustre: Skipped 1 previous similar message [ 1560.858219] Lustre: server umount lustre-OST0000 complete [ 1563.104216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1563.106864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.112111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.404825] Lustre: server umount lustre-MDT0000 complete [ 1568.736058] LustreError: 93915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.741632] LustreError: 93915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1569.628290] LustreError: 93896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943063 with bad export cookie 16363104863452684460 [ 1569.631139] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.632507] LustreError: 93896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1569.745466] Lustre: server umount lustre-MDT0001 complete [ 1573.180983] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1574.250700] Key type lgssc unregistered [ 1574.383395] LNet: 97114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.385833] LNetError: 97114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.395301] LNet: Removed LNI 192.168.202.116@tcp [ 1574.704509] Key type .llcrypt unregistered [ 1574.705666] Key type ._llcrypt unregistered [ 1581.483079] Key type ._llcrypt registered [ 1581.484673] Key type .llcrypt registered [ 1581.529808] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1581.850130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1581.864421] alg: No test for adler32 (adler32-zlib) [ 1582.722104] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1582.812679] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1584.399159] Key type lgssc registered [ 1584.762889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1586.770908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1586.770931] LustreError: 98157:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1586.773305] Lustre: Skipped 1 previous similar message [ 1586.778157] LustreError: 98156:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1586.781154] LustreError: 98157:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1586.783405] LustreError: 98157:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1586.787091] LustreError: 98157:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1586.789639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.959755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.399983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.500809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1591.755815] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1593.390425] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing lsmod [ 1594.342182] LustreError: 99329:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1594.342245] LustreError: 99328:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1594.348712] LustreError: 99329:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1594.351856] LustreError: 99329:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1594.354722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1594.356235] LustreError: 99329:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1594.483566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.871749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1595.874034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.877688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.968382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.969548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.970928] Lustre: Skipped 1 previous similar message [ 1599.975962] Lustre: Skipped 1 previous similar message [ 1600.993372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.997062] Lustre: Skipped 3 previous similar messages [ 1601.497756] Lustre: server umount lustre-MDT0000 complete [ 1602.767160] LustreError: 98170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943096 with bad export cookie 10394498740467436081 [ 1602.770450] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1602.771169] LustreError: 98170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1603.015266] LustreError: 98624:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000043a11695 x1860114134875008/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1603.072739] Lustre: server umount lustre-MDT0001 complete [ 1614.356791] Lustre: server umount lustre-OST0000 complete [ 1617.121960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.277614] LustreError: 100394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.286667] LustreError: 100394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1617.301050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.632662] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1621.586548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.717255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1623.073358] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1624.880326] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1626.044538] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1628.302468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.266424] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1632.040443] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1641.766478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1647.072698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.079309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1647.079746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.084375] Lustre: Skipped 1 previous similar message [ 1649.870061] Lustre: server umount lustre-OST0000 complete [ 1652.193096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.194184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.198496] Lustre: Skipped 2 previous similar messages [ 1657.236805] Lustre: server umount lustre-MDT0000 complete [ 1657.311921] LustreError: 100394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.316700] LustreError: 100394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1658.460358] LustreError: 100376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943152 with bad export cookie 10394498740467436732 [ 1658.462490] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1658.463709] LustreError: 100376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1658.577096] Lustre: server umount lustre-MDT0001 complete [ 1662.067160] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1663.080330] Key type lgssc unregistered [ 1663.201387] LNet: 103589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.204855] LNetError: 103589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.215316] LNet: Removed LNI 192.168.202.116@tcp [ 1663.508943] Key type .llcrypt unregistered [ 1663.510337] Key type ._llcrypt unregistered [ 1673.520192] Key type ._llcrypt registered [ 1673.521521] Key type .llcrypt registered [ 1673.589893] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1673.996535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1674.008484] alg: No test for adler32 (adler32-zlib) [ 1674.889684] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1675.020522] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1676.631171] Key type lgssc registered [ 1676.993662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1679.871878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1679.884294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.111458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.552170] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1685.708640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.925871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1687.553609] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1689.648218] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1691.032604] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1693.747182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1693.918483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.297285] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1698.375836] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1706.227649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1711.238581] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1712.248513] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1716.713747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.714967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1716.722445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1716.726084] LustreError: Skipped 1 previous similar message [ 1719.361494] Lustre: server umount lustre-OST0000 complete [ 1721.826569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1721.827244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.828403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1721.828412] Lustre: Skipped 1 previous similar message [ 1721.844856] Lustre: Skipped 2 previous similar messages [ 1726.948080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1726.951969] Lustre: Skipped 2 previous similar messages [ 1732.066326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.070379] Lustre: Skipped 1 previous similar message [ 1735.135083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1735.195358] Lustre: server umount lustre-MDT0000 complete [ 1736.449609] LustreError: 107182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943230 with bad export cookie 1372440009860095597 [ 1736.451649] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.453068] LustreError: 107182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1736.565429] Lustre: server umount lustre-MDT0001 complete [ 1741.015873] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1742.419296] Key type lgssc unregistered [ 1742.585670] LNet: 108267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.591312] LNetError: 108267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.601651] LNet: Removed LNI 192.168.202.116@tcp [ 1743.026840] Key type .llcrypt unregistered [ 1743.028957] Key type ._llcrypt unregistered [ 1751.419563] Key type ._llcrypt registered [ 1751.420660] Key type .llcrypt registered [ 1751.470208] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1751.834888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.866540] alg: No test for adler32 (adler32-zlib) [ 1752.723747] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1752.801517] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1754.391206] Key type lgssc registered [ 1754.969877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1758.934312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.945285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.172830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.950294] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1765.692730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.819872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1767.305366] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1769.974557] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1771.694929] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1774.740106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1774.893719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.414099] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1779.977279] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1785.133415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1790.437081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1790.437885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.440230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1790.454971] Lustre: Skipped 1 previous similar message [ 1794.321976] Lustre: server umount lustre-OST0000 complete [ 1796.576373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1796.580905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.588587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1796.592579] Lustre: Skipped 1 previous similar message [ 1800.673210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.674246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.793055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.796900] Lustre: Skipped 2 previous similar messages [ 1810.399101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1810.454894] Lustre: server umount lustre-MDT0000 complete [ 1810.912121] LustreError: 109452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1810.916105] LustreError: 109452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1811.601188] LustreError: 111622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943305 with bad export cookie 7661843758988551777 [ 1811.604513] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1811.606512] LustreError: 111622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1811.724896] Lustre: server umount lustre-MDT0001 complete [ 1814.973296] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1815.852414] Key type lgssc unregistered [ 1815.967302] LNet: 112645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.969223] LNetError: 112645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.975284] LNet: Removed LNI 192.168.202.116@tcp [ 1816.229689] Key type .llcrypt unregistered [ 1816.230610] Key type ._llcrypt unregistered [ 1824.260716] Key type ._llcrypt registered [ 1824.261644] Key type .llcrypt registered [ 1824.293513] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1824.571225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.613040] alg: No test for adler32 (adler32-zlib) [ 1825.458770] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1825.533580] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1827.111121] Key type lgssc registered [ 1827.448048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.823525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.829430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.002560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.562854] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1836.391420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.558136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1838.204405] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1840.741945] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1842.343540] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1845.261798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1845.448716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.101420] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1850.246724] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1855.721164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1863.461805] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1864.998552] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1865.002353] Lustre: Skipped 2 previous similar messages [ 1866.940392] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1866.944975] Lustre: Skipped 2 previous similar messages [ 1869.301780] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1869.310550] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1875.612288] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1875.615295] Lustre: Skipped 2 previous similar messages [ 1878.866435] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.16@tcp 192.168.0.[2,10]@tcp [ 1885.005960] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.16/24@tcp in log lustre-MDT0000 [ 1885.011255] Lustre: Skipped 2 previous similar messages [ 1891.859994] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.16/24@tcp [ 1891.863685] Lustre: Skipped 1 previous similar message [ 1896.930668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1896.934991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.938597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1896.944676] Lustre: Skipped 1 previous similar message [ 1900.383455] Lustre: server umount lustre-OST0000 complete [ 1903.072416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1903.077117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.085260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1903.088652] Lustre: Skipped 1 previous similar message [ 1907.169055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.170216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.317834] Lustre: server umount lustre-MDT0000 complete [ 1909.585263] LustreError: 114739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943403 with bad export cookie 6585434882995928191 [ 1909.590134] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.590651] LustreError: 114739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1909.739757] Lustre: server umount lustre-MDT0001 complete [ 1914.808774] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1915.847864] Key type lgssc unregistered [ 1915.979582] LNet: 118717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.982213] LNetError: 118717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.988615] LNet: Removed LNI 192.168.202.116@tcp [ 1916.351438] Key type .llcrypt unregistered [ 1916.352827] Key type ._llcrypt unregistered [ 1930.393720] Key type ._llcrypt registered [ 1930.394671] Key type .llcrypt registered [ 1930.429972] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1930.721051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.726040] alg: No test for adler32 (adler32-zlib) [ 1931.594930] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 1931.683489] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1933.271161] Key type lgssc registered [ 1933.853228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.694444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1937.705674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.896646] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1938.902104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.906491] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.16/24@tcp [ 1940.227663] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1944.186373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.313896] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1944.323842] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.16/24@tcp [ 1944.342395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1946.224380] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1948.556881] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.845298] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1953.016637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1953.183050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.147350] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1956.669421] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1963.431647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 1973.732457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.733262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.741504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.751762] Lustre: Skipped 1 previous similar message [ 1975.888522] Lustre: server umount lustre-OST0000 complete [ 1978.850192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.851204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.856717] Lustre: Skipped 2 previous similar messages [ 1983.732697] Lustre: server umount lustre-MDT0000 complete [ 1983.968757] LustreError: 121366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.977977] LustreError: 121366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1985.528140] LustreError: 120219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943479 with bad export cookie 3638322951636719314 [ 1985.534058] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1985.535777] LustreError: 120219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1985.695252] Lustre: server umount lustre-MDT0001 complete [ 1990.152373] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1991.603358] Key type lgssc unregistered [ 1991.760602] LNet: 123494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.765279] LNetError: 123494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.775551] LNet: Removed LNI 192.168.202.116@tcp [ 1992.225569] Key type .llcrypt unregistered [ 1992.227300] Key type ._llcrypt unregistered [ 2003.017952] Key type ._llcrypt registered [ 2003.019024] Key type .llcrypt registered [ 2003.062242] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2010.209269] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2010.635734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2010.650542] alg: No test for adler32 (adler32-zlib) [ 2011.510989] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 2011.602299] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2013.199165] Key type lgssc registered [ 2013.772756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2029.998951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2030.005971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.113931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2031.124854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2031.173983] Lustre: lustre-MDT0000: new disk, initializing [ 2031.197420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.203460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2032.690351] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2037.238418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.296535] Lustre: 126832:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2037.334478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2037.339320] Lustre: Skipped 1 previous similar message [ 2037.384969] Lustre: lustre-MDT0001: new disk, initializing [ 2037.417503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2037.429518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2037.432819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2039.043571] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2042.925831] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2044.478056] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2047.703934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2047.864194] Lustre: lustre-OST0000: new disk, initializing [ 2047.867827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2047.911897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.282977] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2053.120530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2053.124856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2053.162472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2053.785587] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2055.385094] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2055.449755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2057.140850] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2057.256760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2058.212889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.214100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2058.223379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2058.227148] LustreError: Skipped 1 previous similar message [ 2060.269196] Lustre: server umount lustre-OST0000 complete [ 2063.330371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.330510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2063.332578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2063.332587] Lustre: Skipped 1 previous similar message [ 2063.337670] Lustre: Skipped 2 previous similar messages [ 2068.144825] Lustre: server umount lustre-MDT0000 complete [ 2069.891167] LustreError: 126824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943563 with bad export cookie 11039680889158174480 [ 2069.896938] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.899075] LustreError: 126824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2070.048199] Lustre: server umount lustre-MDT0001 complete [ 2074.088999] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.300347] LustreError: 130641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.308398] LustreError: 130641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2074.332489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.943483] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2079.712859] LustreError: 130642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.024993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.233158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.823182] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2084.340745] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2085.954795] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2089.275051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2089.431325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.873423] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2094.403829] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2095.999417] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2099.137836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2100.425496] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2100.522578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2104.945321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2105.060601] Lustre: lustre-OST0001: new disk, initializing [ 2105.064309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2107.583026] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2111.556471] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2111.997116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2111.999805] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2112.041171] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2112.689613] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2114.374137] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2114.486067] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2116.140548] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d4dc3478800.ost_server_uuid 50 [ 2116.851295] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d4dc3478800.ost_server_uuid in FULL state after 0 sec [ 2122.215269] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2122.216857] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.219186] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2122.219194] Lustre: Skipped 1 previous similar message [ 2122.219781] LustreError: Skipped 1 previous similar message [ 2122.235264] Lustre: Skipped 1 previous similar message [ 2125.386541] Lustre: server umount lustre-OST0001 complete [ 2127.332190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.334283] LustreError: 132233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.335653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2127.335667] Lustre: Skipped 1 previous similar message [ 2127.341128] Lustre: Skipped 1 previous similar message [ 2127.350288] LustreError: 132233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2132.453209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2132.454466] LustreError: 132235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.457098] Lustre: Skipped 2 previous similar messages [ 2132.468561] LustreError: 132235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2133.282442] Lustre: server umount lustre-OST0000 complete [ 2136.544343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2136.548794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.689121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.692801] Lustre: Skipped 4 previous similar messages [ 2149.343130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2149.430327] Lustre: server umount lustre-MDT0000 complete [ 2151.216950] LustreError: 130621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943644 with bad export cookie 11039680889158175362 [ 2151.222592] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.225899] LustreError: 130621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2151.383669] Lustre: server umount lustre-MDT0001 complete [ 2159.442878] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2160.883913] Key type lgssc unregistered [ 2161.042664] LNet: 136507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.047424] LNetError: 136507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.060550] LNet: Removed LNI 192.168.202.116@tcp [ 2161.487341] Key type .llcrypt unregistered [ 2161.489464] Key type ._llcrypt unregistered [ 2179.871632] Key type ._llcrypt registered [ 2179.873368] Key type .llcrypt registered [ 2179.939597] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2187.956969] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2188.510277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.533631] alg: No test for adler32 (adler32-zlib) [ 2189.405084] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 2189.507324] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2191.087158] Key type lgssc registered [ 2191.704589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2208.511379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2208.521437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.666064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2209.686446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2209.738034] Lustre: lustre-MDT0000: new disk, initializing [ 2209.779299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.791818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2211.636692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2217.053968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.107962] Lustre: 140598:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2217.139934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2217.142233] Lustre: Skipped 1 previous similar message [ 2217.187533] Lustre: lustre-MDT0001: new disk, initializing [ 2217.234910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2217.254557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2217.257941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2218.673376] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2222.567523] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2224.178664] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2227.324444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2227.472101] Lustre: lustre-OST0000: new disk, initializing [ 2227.475839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2227.523157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.084142] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2233.349986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2233.355961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2233.404849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2234.124765] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2236.608567] Lustre: Failing over lustre-OST0000 [ 2236.717140] Lustre: server umount lustre-OST0000 complete [ 2250.372220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2250.521414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.527192] Lustre: Skipped 1 previous similar message [ 2250.532517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.541735] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2250.736501] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2252.976711] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2254.624212] Lustre: 138828:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773943732/real 1773943732] req@00000000f7f04969 x1860114771372416/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773943748 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2254.798201] Lustre: Failing over lustre-MDT0000 [ 2254.920773] Lustre: server umount lustre-MDT0000 complete [ 2255.839393] Lustre: 138825:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773943732/real 1773943732] req@00000000731f5211 x1860114771373056/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773943748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2255.844180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.856295] Lustre: 138825:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2255.864116] Lustre: Skipped 1 previous similar message [ 2259.935170] Lustre: 138826:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773943737/real 1773943737] req@00000000093dc1d0 x1860114771374080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773943753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2260.960564] LustreError: 140609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.970770] LustreError: 140609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2265.055084] Lustre: 138826:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773943742/real 1773943742] req@000000007a8fdf9e x1860114771375104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773943758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2266.079970] LustreError: 140610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.084552] LustreError: 140610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2268.116357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.176153] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.180096] Lustre: 140601:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.202.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2268.275412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2268.305120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.315884] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2268.316574] Lustre: lustre-OST0000: Recovery over after 0:18, of 3 clients 3 recovered and 0 were evicted. [ 2268.337713] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2269.850377] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2271.205057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2273.767672] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2273.771502] Lustre: Skipped 1 previous similar message [ 2273.785214] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2273.806573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2278.881168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2278.882095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.886055] LustreError: Skipped 1 previous similar message [ 2278.890866] Lustre: Skipped 1 previous similar message [ 2278.891894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2278.897878] Lustre: Skipped 1 previous similar message [ 2281.173789] Lustre: server umount lustre-OST0000 complete [ 2284.001486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.001890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.002865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.013370] Lustre: Skipped 2 previous similar messages [ 2288.736306] Lustre: server umount lustre-MDT0000 complete [ 2289.121377] LustreError: 141659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.130470] LustreError: 141659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2290.184546] LustreError: 140589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943783 with bad export cookie 12088266403024884193 [ 2290.191632] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.192897] LustreError: 140589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2290.320746] Lustre: server umount lustre-MDT0001 complete [ 2295.206897] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2296.708107] Key type lgssc unregistered [ 2296.880557] LNet: 145830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.884771] LNetError: 145830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.895857] LNet: Removed LNI 192.168.202.116@tcp [ 2297.362355] Key type .llcrypt unregistered [ 2297.364411] Key type ._llcrypt unregistered [ 2319.806925] Key type ._llcrypt registered [ 2319.807971] Key type .llcrypt registered [ 2319.844227] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2320.369375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.388948] alg: No test for adler32 (adler32-zlib) [ 2321.247414] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 2321.349735] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2322.935157] Key type lgssc registered [ 2323.480894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2326.865279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2326.874498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.069540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.777512] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2333.826540] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2336.251232] LustreError: 147358:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2336.257763] LustreError: 147358:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2336.263351] LustreError: 147358:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2336.417837] Lustre: server umount lustre-MDT0000 complete [ 2342.926340] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2346.871542] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2363.502356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.636186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2363.649731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2363.702331] Lustre: lustre-MDT0000: new disk, initializing [ 2363.731605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.741362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2365.458870] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2370.821885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.877819] Lustre: 150508:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2370.902459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2370.904500] Lustre: Skipped 1 previous similar message [ 2370.948106] Lustre: lustre-MDT0001: new disk, initializing [ 2370.989329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.000494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2371.003531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2372.585544] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2376.635879] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2378.358832] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2381.604280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2381.784376] Lustre: lustre-OST0000: new disk, initializing [ 2381.788570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2381.838399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.575326] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2388.726600] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2389.523512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2389.529869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2389.593314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2396.154947] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2396.951830] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:11:29 (1773943889) [ 2458.079385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2458.081352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.084467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.151533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.151580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.153045] Lustre: Skipped 1 previous similar message [ 2461.156228] Lustre: Skipped 1 previous similar message [ 2463.709459] Lustre: server umount lustre-MDT0000 complete [ 2464.757518] LustreError: 150502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773943958 with bad export cookie 5619108644817448955 [ 2464.758582] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2464.761287] LustreError: 150502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2466.271366] LustreError: 150516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.271404] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.275560] LustreError: 150516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2466.278736] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2466.282472] Lustre: Skipped 1 previous similar message [ 2469.279368] LustreError: 153902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.279379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2474.463349] LustreError: 152848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.463400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2479.071142] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2479.151601] Lustre: server umount lustre-MDT0001 complete [ 2487.007984] Lustre: server umount lustre-OST0000 complete [ 2490.837519] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2494.581069] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2510.413527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.564113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2510.583504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2510.627485] Lustre: lustre-MDT0000: new disk, initializing [ 2510.685107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.699353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2512.419917] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2517.609942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.668861] Lustre: 156733:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2517.701299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2517.704475] Lustre: Skipped 1 previous similar message [ 2517.756534] Lustre: lustre-MDT0001: new disk, initializing [ 2517.818801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2517.823596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2519.503410] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2523.085367] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2524.514637] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2526.858361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2526.957797] Lustre: lustre-OST0000: new disk, initializing [ 2526.959781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2526.989488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.991388] Lustre: Skipped 1 previous similar message [ 2529.055749] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2532.317367] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2533.370219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2533.374888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2533.409285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2533.687482] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2533.770703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2535.103167] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2535.188256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2538.465325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2538.466353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.467112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2538.467116] Lustre: Skipped 1 previous similar message [ 2538.467533] LustreError: Skipped 1 previous similar message [ 2538.471644] Lustre: Skipped 1 previous similar message [ 2541.971750] Lustre: server umount lustre-OST0000 complete [ 2543.584684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.585090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.592090] Lustre: Skipped 1 previous similar message [ 2549.270543] Lustre: server umount lustre-MDT0000 complete [ 2550.464057] LustreError: 158474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944044 with bad export cookie 5619108644817607785 [ 2550.467648] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.468705] LustreError: 158474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2550.581060] Lustre: server umount lustre-MDT0001 complete [ 2553.479222] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:14:05 (1773944045) [ 2559.007737] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2562.395296] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2579.246381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.375593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2579.393386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2579.445988] Lustre: lustre-MDT0000: new disk, initializing [ 2579.492526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.500204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2581.712729] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2586.778167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.826691] Lustre: 163046:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2586.858817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2586.861247] Lustre: Skipped 5 previous similar messages [ 2586.941244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2588.355901] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2591.554362] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2592.727540] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2595.149232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.126328] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2600.067631] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2600.423678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2600.436345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2610.656426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2610.658897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.660215] LustreError: Skipped 1 previous similar message [ 2613.462347] Lustre: server umount lustre-OST0000 complete [ 2620.896502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.901374] Lustre: Skipped 5 previous similar messages [ 2629.087102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2629.139593] Lustre: server umount lustre-MDT0000 complete [ 2630.425550] LustreError: 163037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944124 with bad export cookie 5619108644817608653 [ 2630.428913] LustreError: 163037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2630.429858] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.557335] Lustre: server umount lustre-MDT0001 complete [ 2633.583975] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:15:25 (1773944125) [ 2639.539428] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2642.988836] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2656.453257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.572845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2656.585965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2656.589239] Lustre: Skipped 3 previous similar messages [ 2656.630091] Lustre: lustre-MDT0000: new disk, initializing [ 2656.632444] Lustre: Skipped 2 previous similar messages [ 2656.660463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.662864] Lustre: Skipped 2 previous similar messages [ 2656.669278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2656.672366] Lustre: Skipped 2 previous similar messages [ 2658.119322] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2662.517147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.563190] Lustre: 168977:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2662.670169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2664.248890] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2667.785542] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2669.230956] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2672.058309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2674.506245] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2677.979937] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2678.306204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2688.480686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.482995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.483544] LustreError: Skipped 1 previous similar message [ 2688.487501] Lustre: Skipped 3 previous similar messages [ 2688.488625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.495151] Lustre: Skipped 3 previous similar messages [ 2691.552704] Lustre: server umount lustre-OST0000 complete [ 2699.439349] Lustre: server umount lustre-MDT0000 complete [ 2700.906195] LustreError: 168968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944194 with bad export cookie 5619108644817609731 [ 2700.909985] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.910477] LustreError: 168968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2701.045066] Lustre: server umount lustre-MDT0001 complete [ 2705.060046] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2706.135183] Key type lgssc unregistered [ 2706.281811] LNet: 172454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2706.284805] LNetError: 172454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2706.294353] LNet: Removed LNI 192.168.202.116@tcp [ 2706.593929] Key type .llcrypt unregistered [ 2706.595340] Key type ._llcrypt unregistered [ 2716.214492] Key type ._llcrypt registered [ 2716.216581] Key type .llcrypt registered [ 2716.266665] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2716.611777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2716.652517] alg: No test for adler32 (adler32-zlib) [ 2717.508816] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 2717.603285] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2719.191161] Key type lgssc registered [ 2719.577292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2722.619513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2722.627829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.822290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2725.185872] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2728.184800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.316327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.711520] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2731.624802] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2732.953670] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2735.477684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2735.598261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.558904] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2739.492381] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2748.902786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2763.231955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2763.235737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.241371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.257050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2764.258468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.258793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.267766] Lustre: Skipped 1 previous similar message [ 2767.784760] Lustre: server umount lustre-OST0000 complete [ 2769.376463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.376918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2769.378214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2769.380752] Lustre: Skipped 1 previous similar message [ 2774.496145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2774.498215] Lustre: Skipped 2 previous similar messages [ 2775.393487] Lustre: server umount lustre-MDT0000 complete [ 2776.842720] LustreError: 173811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944270 with bad export cookie 7791807173196209578 [ 2776.846157] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.849519] LustreError: 173811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2776.981825] Lustre: server umount lustre-MDT0001 complete [ 2780.859979] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2781.987886] Key type lgssc unregistered [ 2782.119949] LNet: 177028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2782.123550] LNetError: 177028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2782.131276] LNet: Removed LNI 192.168.202.116@tcp [ 2782.485641] Key type .llcrypt unregistered [ 2782.487231] Key type ._llcrypt unregistered [ 2792.492577] Key type ._llcrypt registered [ 2792.493842] Key type .llcrypt registered [ 2792.544365] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2792.869119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2792.879837] alg: No test for adler32 (adler32-zlib) [ 2793.764288] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 2793.846290] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2795.431499] Key type lgssc registered [ 2795.874845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2798.845608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2798.852823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.010189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.424446] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2805.072255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.258970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.807566] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2808.881901] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2810.226928] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2812.834341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2812.960769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.953950] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2817.014057] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2826.280583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2831.331881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.332492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2831.333485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2831.338124] Lustre: Skipped 1 previous similar message [ 2833.910495] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 2833.913626] Lustre: Skipped 1 previous similar message [ 2836.456957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2836.460474] Lustre: Skipped 1 previous similar message [ 2837.201443] Lustre: server umount lustre-OST0000 complete [ 2839.260450] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d4dc43eb000.ost_server_uuid 50 [ 2839.852500] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2849.995652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.996371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.000822] Lustre: Skipped 1 previous similar message [ 2855.394409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2855.397958] Lustre: Skipped 2 previous similar messages [ 2856.152707] Lustre: server umount lustre-MDT0000 complete [ 2857.452200] LustreError: 178385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944351 with bad export cookie 7202408962195453130 [ 2857.455462] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.455819] LustreError: 178385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2857.563531] Lustre: server umount lustre-MDT0001 complete [ 2860.458287] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:19:12 (1773944352) [ 2862.841240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.994303] LustreError: 181882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.000924] LustreError: 181882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2863.019077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.376294] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2867.242952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.369596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2868.617952] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2870.604179] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2871.891706] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2874.108679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.949588] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2877.605641] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2879.779809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2879.855961] Lustre: lustre-OST0001: new disk, initializing [ 2879.857674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2879.888186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2879.890482] Lustre: Skipped 1 previous similar message [ 2881.677015] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2882.112814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2882.116348] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2882.146144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2884.829625] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2887.139584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.144550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2891.832926] Lustre: server umount lustre-OST0000 complete [ 2893.284210] LustreError: 183476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.290044] LustreError: 183476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2893.687207] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2893.761241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2895.056842] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2895.130970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2895.331860] LustreError: 184802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.403716] LustreError: 183476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.409089] LustreError: 183476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2903.522888] LustreError: 183986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.527720] LustreError: 183986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2907.615796] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2907.618754] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.619614] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2907.623188] Lustre: Skipped 1 previous similar message [ 2907.625085] Lustre: Skipped 3 previous similar messages [ 2911.245372] Lustre: server umount lustre-OST0001 complete [ 2913.760695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2913.760979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.762800] LustreError: Skipped 1 previous similar message [ 2913.766043] Lustre: Skipped 1 previous similar message [ 2918.617627] Lustre: server umount lustre-MDT0000 complete [ 2918.880025] LustreError: 181883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.885769] LustreError: 181883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2919.869131] LustreError: 181863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944413 with bad export cookie 7202408962195454187 [ 2919.872462] LustreError: 181863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2919.873033] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2919.991862] Lustre: server umount lustre-MDT0001 complete [ 2928.230394] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:20:20 (1773944420) [ 2930.318091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2930.408622] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2930.423466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2930.489804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.769926] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2934.607125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.639726] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2934.652186] Lustre: 187655:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2936.086139] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2937.957647] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2939.194255] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2941.387080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.449520] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2942.287493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2943.330627] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2945.116616] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2947.381692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.414982] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2949.319101] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2950.229481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2951.088185] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2952.672137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2952.672405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.675736] LustreError: Skipped 1 previous similar message [ 2952.680746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2952.684152] Lustre: Skipped 5 previous similar messages [ 2958.012319] Lustre: server umount lustre-OST0000 complete [ 2960.354488] LustreError: 189439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.359406] LustreError: 189439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2970.593463] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2970.595598] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.595724] LustreError: Skipped 1 previous similar message [ 2970.600424] Lustre: Skipped 1 previous similar message [ 2974.682637] Lustre: server umount lustre-OST0001 complete [ 2980.832833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2982.108801] Lustre: server umount lustre-MDT0000 complete [ 2983.380203] LustreError: 188382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944476 with bad export cookie 7202408962195455349 [ 2983.382659] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.383249] LustreError: 188382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2983.488020] Lustre: server umount lustre-MDT0001 complete [ 2991.203743] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:21:23 (1773944483) [ 2996.132121] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2999.142848] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3009.978785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.069583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3010.079876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3010.117977] Lustre: lustre-MDT0000: new disk, initializing [ 3010.145103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.147974] Lustre: Skipped 3 previous similar messages [ 3010.155149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3011.511539] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3015.663118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.703964] Lustre: 195223:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3015.727365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3015.729889] Lustre: Skipped 1 previous similar message [ 3015.766753] Lustre: lustre-MDT0001: new disk, initializing [ 3015.807898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3015.811581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3017.108809] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3020.293358] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3021.630837] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3023.991501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.096112] Lustre: lustre-OST0000: new disk, initializing [ 3024.098294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3026.041715] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3027.257575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3027.261357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3027.297316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3029.151292] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3030.601813] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3030.692209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3032.105222] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3032.190789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3037.664561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3037.666850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.667674] LustreError: Skipped 1 previous similar message [ 3037.672461] Lustre: Skipped 2 previous similar messages [ 3037.673158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3037.678529] Lustre: Skipped 8 previous similar messages [ 3038.997518] Lustre: server umount lustre-OST0000 complete [ 3046.555162] Lustre: server umount lustre-MDT0000 complete [ 3047.904090] LustreError: 195236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.909030] LustreError: 195236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3048.015406] LustreError: 195218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944541 with bad export cookie 7202408962195457113 [ 3048.020046] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.020733] LustreError: 195218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3048.167371] Lustre: server umount lustre-MDT0001 complete [ 3051.333025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.976047] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3056.100556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.644686] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3059.616603] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3061.022486] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3063.416235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.467693] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3067.358059] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3068.806720] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3078.082151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3079.470779] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3079.554169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3081.183812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3081.185926] LustreError: Skipped 1 previous similar message [ 3086.353823] Lustre: server umount lustre-OST0000 complete [ 3088.473352] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3089.853618] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3113.539775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.652780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.656327] Lustre: Skipped 5 previous similar messages [ 3115.526926] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3117.333953] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3117.893278] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3120.904251] Lustre: server umount lustre-OST0000 complete [ 3123.167812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.172605] Lustre: Skipped 5 previous similar messages [ 3128.285598] Lustre: server umount lustre-MDT0000 complete [ 3129.311936] LustreError: 201976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.316954] LustreError: 201976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3129.501267] LustreError: 199750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944623 with bad export cookie 7202408962195457981 [ 3129.503960] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.507271] LustreError: 199750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3129.616276] Lustre: server umount lustre-MDT0001 complete [ 3132.371393] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:23:44 (1773944624) [ 3134.536531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.995859] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3139.050681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.568029] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3142.425438] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3143.652279] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3145.862160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3147.807033] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3149.650342] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3151.119940] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3161.385169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3162.759891] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3162.841766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3165.258924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3165.333886] Lustre: lustre-OST0001: new disk, initializing [ 3165.336408] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3167.325933] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3170.398607] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3171.326593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3171.332411] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3171.364653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3171.775947] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3171.850624] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3173.200708] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3173.284698] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3176.416926] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3176.419758] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3176.419839] LustreError: Skipped 3 previous similar messages [ 3176.421888] Lustre: Skipped 9 previous similar messages [ 3180.032268] Lustre: server umount lustre-OST0001 complete [ 3182.036183] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3182.109570] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3183.426949] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3183.509404] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3207.049473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3209.005497] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3210.881241] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3211.435346] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3218.166172] Lustre: server umount lustre-OST0001 complete [ 3236.109916] Lustre: server umount lustre-OST0000 complete [ 3243.477181] Lustre: server umount lustre-MDT0000 complete [ 3244.636995] LustreError: 204517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944738 with bad export cookie 7202408962195458912 [ 3244.637966] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3244.640785] LustreError: 204517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3250.804486] Lustre: server umount lustre-MDT0001 complete [ 3258.266071] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:25:50 (1773944750) [ 3260.344430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.437040] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3260.446736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3260.518699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3260.520972] Lustre: Skipped 5 previous similar messages [ 3261.768712] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3264.474152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.508352] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3264.518524] Lustre: 212544:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3265.933470] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3267.604105] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3268.700878] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3270.705055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3270.764639] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3272.601848] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3274.288413] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3283.149481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3283.172299] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3284.933294] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3286.592537] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3289.165618] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3290.481123] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3292.600357] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3294.009021] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3294.083326] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3295.163270] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d4dc9323000.ost_server_uuid 50 [ 3295.625383] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d4dc9323000.ost_server_uuid in FULL state after 0 sec [ 3295.932554] Lustre: Permanently deactivating lustre-OST0001 [ 3295.936816] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3296.635670] Lustre: Permanently reactivating lustre-OST0001 [ 3301.859019] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.863349] Lustre: Skipped 8 previous similar messages [ 3303.553563] Lustre: server umount lustre-OST0001 complete [ 3306.978036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3306.979048] LustreError: 214732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3306.979995] LustreError: Skipped 4 previous similar messages [ 3306.984487] LustreError: 214732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 3319.263129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3319.317175] Lustre: server umount lustre-OST0000 complete [ 3326.746170] Lustre: server umount lustre-MDT0000 complete [ 3327.932141] LustreError: 212536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944821 with bad export cookie 7202408962195460172 [ 3327.934360] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.937365] LustreError: 212536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3328.061172] Lustre: server umount lustre-MDT0001 complete [ 3335.415648] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:27:07 (1773944827) [ 3337.709536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.794364] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3337.803995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3337.806863] Lustre: Skipped 5 previous similar messages [ 3338.966338] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3341.526868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.559775] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3341.569626] Lustre: 218664:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3342.937043] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3344.625545] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3345.750450] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3347.830522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3347.903822] Lustre: Permanently deactivating lustre-OST0000 [ 3347.909096] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3349.756528] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3351.449352] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3353.534180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3354.871706] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3354.910205] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3355.381561] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3357.052542] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3357.655205] Lustre: Permanently reactivating lustre-OST0000 [ 3360.567545] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3360.592600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3371.716494] Lustre: server umount lustre-OST0001 complete [ 3387.359120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3387.410708] Lustre: server umount lustre-OST0000 complete [ 3394.905235] Lustre: server umount lustre-MDT0000 complete [ 3396.037095] LustreError: 218655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944889 with bad export cookie 7202408962195461831 [ 3396.040208] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3396.040780] LustreError: 218655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3396.147614] Lustre: server umount lustre-MDT0001 complete [ 3399.365947] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3400.326539] Key type lgssc unregistered [ 3400.459348] LNet: 223038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3400.462062] LNetError: 223038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3400.469711] LNet: Removed LNI 192.168.202.116@tcp [ 3400.753303] Key type .llcrypt unregistered [ 3400.754889] Key type ._llcrypt unregistered [ 3409.063339] Key type ._llcrypt registered [ 3409.064807] Key type .llcrypt registered [ 3409.104815] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3409.400719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3409.412520] alg: No test for adler32 (adler32-zlib) [ 3410.263137] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 3410.338246] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3411.919149] Key type lgssc registered [ 3412.281017] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3415.505191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3415.511068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.674404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.961437] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3420.854063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.894276] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3420.904494] Lustre: 224454:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3420.911707] Lustre: Permanently deactivating lustre-MDT0001 [ 3420.917792] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3421.027912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3422.331264] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3423.340223] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3423.342650] LustreError: 224497:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3423.346687] LustreError: 224497:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3423.350949] LustreError: 224497:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3424.073520] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3425.272460] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3427.570460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3427.672199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3429.468305] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3430.754675] LustreError: 226078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3430.759867] LustreError: 226078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3430.766169] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3431.200434] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3433.298171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3433.362309] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3435.057633] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3436.205791] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3436.673183] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3442.343702] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3442.343812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3447.872853] Lustre: Permanently reactivating lustre-MDT0001 [ 3447.879233] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3447.882062] Lustre: Skipped 2 previous similar messages [ 3455.464952] Lustre: lustre-MDT0001: Client 979ce400-169b-432a-9508-be26c3651bc9 (at 192.168.202.16@tcp) reconnecting [ 3457.245587] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3457.819161] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3459.454827] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3459.521798] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3460.731596] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3460.811354] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3461.551370] Lustre: Permanently deactivating lustre-MDT0001 [ 3461.555894] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3461.557866] Lustre: Skipped 2 previous similar messages [ 3463.485407] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3467.232103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3467.234559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3467.238548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.256112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.257385] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.263420] Lustre: Skipped 2 previous similar messages [ 3472.160572] Lustre: server umount lustre-MDT0000 complete [ 3473.376699] LustreError: 224465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.381733] LustreError: 224465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3473.411449] LustreError: 224446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773944967 with bad export cookie 16680199829664432411 [ 3473.415532] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3473.416618] LustreError: 224446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3473.530544] Lustre: server umount lustre-MDT0001 complete [ 3480.890150] Lustre: server umount lustre-OST0000 complete [ 3484.461686] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:29:36 (1773944976) [ 3487.668831] Lustre: server umount lustre-OST0001 complete [ 3489.472331] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3492.052267] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3502.919867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.999357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3503.003324] Lustre: Skipped 2 previous similar messages [ 3503.013211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3503.051199] Lustre: lustre-MDT0000: new disk, initializing [ 3503.075267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.082547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3504.257042] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3508.052510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.090435] Lustre: 232119:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3508.108722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3508.110351] Lustre: Skipped 1 previous similar message [ 3508.163825] Lustre: lustre-MDT0001: new disk, initializing [ 3508.202147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3508.205896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3509.419949] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3512.289474] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3513.355760] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3515.455556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.550477] Lustre: lustre-OST0000: new disk, initializing [ 3515.552362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3515.576663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.579066] Lustre: Skipped 1 previous similar message [ 3517.283342] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3520.309508] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3521.028515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3521.033593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3521.071282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3526.124085] LustreError: 232125:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3530.441555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3530.506064] Lustre: lustre-OST0001: new disk, initializing [ 3530.508166] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3532.512877] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3535.713527] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3536.215152] LustreError: 232125:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3536.225635] LustreError: 232124:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3539.326337] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3539.960977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3539.963930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3539.990087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3546.319063] LustreError: 232124:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3548.143126] Lustre: lustre-OST0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 3548.144919] Lustre: Skipped 3 previous similar messages [ 3550.181511] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.183261] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.184333] LustreError: Skipped 1 previous similar message [ 3550.189210] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3550.191778] Lustre: Skipped 1 previous similar message [ 3553.021755] Lustre: server umount lustre-OST0001 complete [ 3553.252467] LustreError: 233848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3555.299661] LustreError: 233848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3555.304306] LustreError: 233848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3560.419554] LustreError: 233848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.425553] LustreError: 233848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3565.538637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.541657] Lustre: Skipped 1 previous similar message [ 3565.543236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3565.545248] Lustre: Skipped 1 previous similar message [ 3570.661560] LustreError: 233847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.666541] LustreError: 233847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3575.778181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.780198] Lustre: Skipped 3 previous similar messages [ 3579.359126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3579.412984] Lustre: server umount lustre-OST0000 complete [ 3580.896564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.901191] Lustre: Skipped 2 previous similar messages [ 3586.904046] Lustre: server umount lustre-MDT0000 complete [ 3588.153363] LustreError: 232112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945081 with bad export cookie 16680199829664435218 [ 3588.156402] LustreError: 232112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3588.156920] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3594.354235] Lustre: server umount lustre-MDT0001 complete [ 3598.040434] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3598.936166] Key type lgssc unregistered [ 3599.063324] LNet: 236761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3599.065587] LNetError: 236761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3599.073269] LNet: Removed LNI 192.168.202.116@tcp [ 3599.336572] Key type .llcrypt unregistered [ 3599.337340] Key type ._llcrypt unregistered [ 3612.795841] Key type ._llcrypt registered [ 3612.797547] Key type .llcrypt registered [ 3612.834249] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3613.184786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3613.209112] alg: No test for adler32 (adler32-zlib) [ 3614.087110] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 3614.187333] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3615.775175] Key type lgssc registered [ 3616.171165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3618.666588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3618.672829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.757416] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3619.763401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3619.817210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3620.977536] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3623.742779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.778422] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3623.789165] Lustre: 238928:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.879697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3625.189226] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3626.910040] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3628.068845] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3630.275617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.342171] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3630.406947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.317794] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3634.101052] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3635.208295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3681.250376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3681.251556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.254192] LustreError: Skipped 1 previous similar message [ 3681.255226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3681.258213] Lustre: Skipped 1 previous similar message [ 3681.261920] Lustre: Skipped 1 previous similar message [ 3684.507363] Lustre: server umount lustre-OST0000 complete [ 3685.950238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3690.206546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3690.318805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.218896] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3693.931351] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3700.580488] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3700.583872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3700.584033] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3700.589299] Lustre: Skipped 1 previous similar message [ 3731.425714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3731.427209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.427652] LustreError: Skipped 1 previous similar message [ 3731.432061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3731.434201] Lustre: Skipped 1 previous similar message [ 3733.137010] Lustre: server umount lustre-OST0000 complete [ 3736.544797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.545149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3736.548537] Lustre: Skipped 1 previous similar message [ 3736.552716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.554637] Lustre: Skipped 1 previous similar message [ 3740.500896] Lustre: server umount lustre-MDT0000 complete [ 3741.738703] LustreError: 238918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945235 with bad export cookie 6343009099664839266 [ 3741.742591] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3741.742649] LustreError: 238918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3741.886776] Lustre: server umount lustre-MDT0001 complete [ 3745.522504] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3746.612374] Key type lgssc unregistered [ 3746.753468] LNet: 243875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3746.755766] LNetError: 243875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3746.762435] LNet: Removed LNI 192.168.202.116@tcp [ 3747.094135] Key type .llcrypt unregistered [ 3747.095312] Key type ._llcrypt unregistered [ 3756.077771] Key type ._llcrypt registered [ 3756.079020] Key type .llcrypt registered [ 3756.118515] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3756.590038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3756.621653] alg: No test for adler32 (adler32-zlib) [ 3757.477290] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 3757.559345] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3759.143116] Key type lgssc registered [ 3759.489657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3762.074843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3762.081337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.239266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.450176] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3767.187930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.307299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3768.631103] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3770.386113] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3771.575509] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3773.812198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3773.926223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.753188] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3777.517061] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3788.263976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3802.592078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.594992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.600049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.618113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.619915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.620140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.627983] Lustre: Skipped 1 previous similar message [ 3806.868911] Lustre: server umount lustre-OST0000 complete [ 3808.223960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3808.227759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.234338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.856215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.858471] Lustre: Skipped 2 previous similar messages [ 3814.366097] Lustre: server umount lustre-MDT0000 complete [ 3815.676346] LustreError: 245232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945309 with bad export cookie 7688237562808015687 [ 3815.678753] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3815.680458] LustreError: 245232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3815.801383] Lustre: server umount lustre-MDT0001 complete [ 3819.585887] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3820.614625] Key type lgssc unregistered [ 3820.744469] LNet: 249221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3820.746980] LNetError: 249221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3820.758393] LNet: Removed LNI 192.168.202.116@tcp [ 3821.067954] Key type .llcrypt unregistered [ 3821.069406] Key type ._llcrypt unregistered [ 3827.906123] Key type ._llcrypt registered [ 3827.907596] Key type .llcrypt registered [ 3827.947749] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3828.422199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3828.428077] alg: No test for adler32 (adler32-zlib) [ 3829.279077] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 3829.364139] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3830.943143] Key type lgssc registered [ 3831.291961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3833.870386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3833.877323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.039843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.342785] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3839.129259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.256559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3840.556525] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3842.374960] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3843.579026] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3845.912197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3846.028512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3848.010444] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3849.837768] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3861.419452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3869.663965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3869.666216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.669856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.714369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.714910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.715414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3875.357505] Lustre: server umount lustre-OST0000 complete [ 3876.832455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3876.833092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3876.839379] Lustre: Skipped 1 previous similar message [ 3876.840854] Lustre: Skipped 1 previous similar message [ 3881.952295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3881.956279] Lustre: Skipped 2 previous similar messages [ 3882.838105] Lustre: server umount lustre-MDT0000 complete [ 3884.119176] LustreError: 252018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945377 with bad export cookie 16287554111464747018 [ 3884.123537] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3884.125666] LustreError: 252018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3884.252579] Lustre: server umount lustre-MDT0001 complete [ 3888.146921] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3889.252458] Key type lgssc unregistered [ 3889.392510] LNet: 253747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3889.396092] LNetError: 253747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3889.407369] LNet: Removed LNI 192.168.202.116@tcp [ 3889.717822] Key type .llcrypt unregistered [ 3889.719294] Key type ._llcrypt unregistered [ 3899.200261] Key type ._llcrypt registered [ 3899.202158] Key type .llcrypt registered [ 3899.247357] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3899.651059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3899.685121] alg: No test for adler32 (adler32-zlib) [ 3900.548671] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 3900.637112] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3902.223219] Key type lgssc registered [ 3902.633632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3905.409873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3905.416273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.608534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.921741] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3910.905052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.059482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3912.426325] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3914.409905] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3915.676145] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3918.089158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3918.226329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.197586] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3922.097607] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3931.559292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3941.857883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3941.858814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.861741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3941.866228] Lustre: Skipped 1 previous similar message [ 3943.503613] Lustre: server umount lustre-OST0000 complete [ 3946.977092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.977575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3946.977801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.977806] Lustre: Skipped 1 previous similar message [ 3946.980192] Lustre: Skipped 1 previous similar message [ 3946.986800] LustreError: Skipped 1 previous similar message [ 3950.935791] Lustre: server umount lustre-MDT0000 complete [ 3952.096087] LustreError: 255122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.103239] LustreError: 255122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3952.242478] LustreError: 256735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945445 with bad export cookie 17621524856083115847 [ 3952.245874] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3952.246122] LustreError: 256735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3952.376486] Lustre: server umount lustre-MDT0001 complete [ 3956.148954] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3957.271323] Key type lgssc unregistered [ 3957.424470] LNet: 259091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3957.426966] LNetError: 259091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3957.434385] LNet: Removed LNI 192.168.202.116@tcp [ 3957.762593] Key type .llcrypt unregistered [ 3957.763945] Key type ._llcrypt unregistered [ 3965.107626] Key type ._llcrypt registered [ 3965.109018] Key type .llcrypt registered [ 3965.157074] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3965.591885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3965.636438] alg: No test for adler32 (adler32-zlib) [ 3966.494764] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 3966.580590] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3968.159151] Key type lgssc registered [ 3968.553290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3971.199371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3971.206569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.388304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.725111] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3976.641171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.778429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3978.144508] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3979.888211] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3981.019394] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3983.200260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3983.328058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.129037] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3986.924874] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3997.671441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4007.905343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4007.905558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.907315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4007.907892] LustreError: Skipped 1 previous similar message [ 4007.910712] Lustre: Skipped 1 previous similar message [ 4012.499394] Lustre: server umount lustre-OST0000 complete [ 4017.631940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4017.635578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.642087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4017.644930] Lustre: Skipped 1 previous similar message [ 4023.264415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.266226] Lustre: Skipped 3 previous similar messages [ 4027.871102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4027.927856] Lustre: server umount lustre-MDT0000 complete [ 4028.384468] LustreError: 260299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.389062] LustreError: 260299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4029.139287] LustreError: 260255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945522 with bad export cookie 2885837888326227779 [ 4029.142551] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.143253] LustreError: 260255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4029.273704] Lustre: server umount lustre-MDT0001 complete [ 4032.803695] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4033.874064] Key type lgssc unregistered [ 4034.002419] LNet: 263655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4034.005221] LNetError: 263655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4034.011387] LNet: Removed LNI 192.168.202.116@tcp [ 4034.312771] Key type .llcrypt unregistered [ 4034.313788] Key type ._llcrypt unregistered [ 4037.958582] Key type ._llcrypt registered [ 4037.960198] Key type .llcrypt registered [ 4038.005962] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4043.883619] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4050.069743] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4050.497292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4050.526326] alg: No test for adler32 (adler32-zlib) [ 4051.403831] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 4051.497129] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4053.087153] Key type lgssc registered [ 4053.430368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4064.705507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4064.710127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4065.801313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4065.814395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4065.855636] Lustre: lustre-MDT0000: new disk, initializing [ 4065.879352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4065.886321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4067.128471] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4071.218032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.264091] Lustre: 267127:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4071.312704] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4071.317479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4071.319263] Lustre: Skipped 1 previous similar message [ 4071.360256] Lustre: lustre-MDT0001: new disk, initializing [ 4071.398865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4071.414827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4071.418754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4072.773501] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4075.750946] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4076.899916] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4079.059534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4079.166402] Lustre: lustre-OST0000: new disk, initializing [ 4079.168180] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4079.194177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.979066] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4084.009649] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4085.372295] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4086.264955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4086.268533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4086.304888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4086.461636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4087.726142] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4087.795533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4091.360854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4091.363189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.364097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4091.366576] Lustre: Skipped 1 previous similar message [ 4094.547885] Lustre: server umount lustre-OST0000 complete [ 4096.481574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.482664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4096.486242] Lustre: Skipped 1 previous similar message [ 4096.488286] Lustre: Skipped 1 previous similar message [ 4101.600584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4101.602885] Lustre: Skipped 2 previous similar messages [ 4106.720717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4106.722811] Lustre: Skipped 1 previous similar message [ 4110.303075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4110.371938] Lustre: server umount lustre-MDT0000 complete [ 4111.592372] LustreError: 267120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945605 with bad export cookie 5271200786803110580 [ 4111.595690] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.596875] LustreError: 267120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4111.720812] Lustre: server umount lustre-MDT0001 complete [ 4114.546981] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:40:06 (1773945606) [ 4116.704340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.875498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.111214] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4120.813070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.957990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4122.237091] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4123.962180] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4125.211793] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4127.453819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.358141] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4131.083294] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4151.263787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4151.265781] LustreError: Skipped 1 previous similar message [ 4151.266876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.270213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4152.290988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4155.536829] Lustre: server umount lustre-OST0000 complete [ 4156.895750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4156.899212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4156.905095] Lustre: Skipped 1 previous similar message [ 4167.649331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4167.651145] Lustre: Skipped 8 previous similar messages [ 4171.231115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4171.291995] Lustre: server umount lustre-MDT0000 complete [ 4172.549188] LustreError: 271131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945666 with bad export cookie 5271200786803111462 [ 4172.552284] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.553966] LustreError: 271131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4172.670591] Lustre: server umount lustre-MDT0001 complete [ 4176.281322] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4177.412215] Key type lgssc unregistered [ 4177.535436] LNet: 274351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4177.538033] LNetError: 274351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4177.547309] LNet: Removed LNI 192.168.202.116@tcp [ 4177.901191] Key type .llcrypt unregistered [ 4177.902164] Key type ._llcrypt unregistered [ 4186.704450] Key type ._llcrypt registered [ 4186.705529] Key type .llcrypt registered [ 4186.745911] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4192.939660] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4193.337659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4193.378263] alg: No test for adler32 (adler32-zlib) [ 4194.232499] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 4194.330120] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4195.919222] Key type lgssc registered [ 4196.315404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4211.768709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4211.777040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.873605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4212.884670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4212.928405] Lustre: lustre-MDT0000: new disk, initializing [ 4212.956382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.962347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4214.305666] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4218.477141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.517551] Lustre: 278114:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4218.541099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4218.542858] Lustre: Skipped 1 previous similar message [ 4218.585093] Lustre: lustre-MDT0001: new disk, initializing [ 4218.619722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4218.632656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4218.635629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4219.908350] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4223.085985] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4224.266091] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4226.659668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4226.786172] Lustre: lustre-OST2710: new disk, initializing [ 4226.789125] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4226.821448] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4227.582511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4227.587900] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4227.648161] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4228.813299] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4232.039849] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4234.327663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.392913] Lustre: lustre-OST03e8: new disk, initializing [ 4234.394751] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4234.426611] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4236.399931] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4238.267384] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4240.963620] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4241.042360] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4242.385501] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4242.465168] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4243.864570] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4243.974411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4243.978760] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4244.017740] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4244.970849] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4246.329488] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4246.420319] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4249.058488] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.060504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.062089] Lustre: Skipped 1 previous similar message [ 4253.607423] Lustre: server umount lustre-MDT0000 complete [ 4254.861263] LustreError: 278106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945748 with bad export cookie 536134761799735522 [ 4254.864206] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4254.865409] LustreError: 278106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4254.982655] Lustre: server umount lustre-MDT0001 complete [ 4265.483338] Lustre: server umount lustre-OST2710 complete [ 4277.026124] Lustre: server umount lustre-OST03e8 complete [ 4283.081494] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4286.259077] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4298.179701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.277317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4298.288810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4298.332086] Lustre: lustre-MDT0000: new disk, initializing [ 4298.359417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.367325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4299.690587] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4303.820440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.863134] Lustre: 285829:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4303.964616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4305.285828] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4308.449688] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4309.696888] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4312.060452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.175109] Lustre: lustre-OST0000: new disk, initializing [ 4312.176479] Lustre: Skipped 1 previous similar message [ 4312.178321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4312.179965] Lustre: Skipped 2 previous similar messages [ 4312.207539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.209738] Lustre: Skipped 1 previous similar message [ 4313.276727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4313.280316] Lustre: Skipped 1 previous similar message [ 4313.282804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4313.314524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4314.122543] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4317.318189] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4318.704632] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4318.776160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4320.131871] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4320.206724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4323.809164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4323.810440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.811854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4323.815656] Lustre: Skipped 3 previous similar messages [ 4323.817793] Lustre: Skipped 3 previous similar messages [ 4328.930405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4328.932461] Lustre: Skipped 2 previous similar messages [ 4334.051153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.053059] Lustre: Skipped 1 previous similar message [ 4335.071148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4335.132253] Lustre: server umount lustre-OST0000 complete [ 4339.168943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.169488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.172256] Lustre: Skipped 1 previous similar message [ 4339.174905] Lustre: Skipped 1 previous similar message [ 4342.618177] Lustre: server umount lustre-MDT0000 complete [ 4343.883345] LustreError: 285824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945837 with bad export cookie 536134761799737125 [ 4343.886597] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4343.886969] LustreError: 285824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4344.011694] Lustre: server umount lustre-MDT0001 complete [ 4346.819813] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:43:59 (1773945839) [ 4347.335754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4347.907515] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:44:00 (1773945840) [ 4348.919881] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4358.025663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4358.129585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4358.138288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4358.209531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.621843] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4362.567327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4362.601522] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4362.613380] Lustre: 291263:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4364.055715] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4365.860527] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4367.102406] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4380.439717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4380.516341] Lustre: Denying initial registration attempt from nid 192.168.202.116@tcp, specified as failover [ 4380.519978] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4380.523193] LustreError: 292875:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 4380.525764] LustreError: 292875:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4380.527648] LustreError: 292875:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4380.550331] Lustre: server umount lustre-OST0000 complete [ 4380.551726] LustreError: 292875:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4383.199943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4383.202337] LustreError: Skipped 1 previous similar message [ 4383.203445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.207270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.023115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4397.101553] Lustre: server umount lustre-MDT0000 complete [ 4398.353327] LustreError: 291255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945891 with bad export cookie 536134761799737993 [ 4398.356300] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4398.357768] LustreError: 291255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4398.473274] Lustre: server umount lustre-MDT0001 complete [ 4402.013418] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4403.053429] Key type lgssc unregistered [ 4403.176436] LNet: 293975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4403.179853] LNetError: 293975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4403.192370] LNet: Removed LNI 192.168.202.116@tcp [ 4403.476350] Key type .llcrypt unregistered [ 4403.478203] Key type ._llcrypt unregistered [ 4407.112285] Key type ._llcrypt registered [ 4407.113369] Key type .llcrypt registered [ 4407.150184] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4407.480915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4407.523907] alg: No test for adler32 (adler32-zlib) [ 4408.373599] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 4408.466868] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4410.047139] Key type lgssc registered [ 4410.406284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4444.826159] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4447.832247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4447.837521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4448.936848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4448.944831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4449.011906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.277966] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4453.133709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4453.164631] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4453.176159] Lustre: 296687:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4453.274201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4454.533547] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4456.478335] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4457.847689] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4471.183182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4471.280627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4471.373051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.330051] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4475.054244] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4475.787573] Lustre: server umount lustre-OST0000 complete [ 4477.943066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.944805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.946302] Lustre: Skipped 1 previous similar message [ 4477.948996] Lustre: Skipped 1 previous similar message [ 4483.041001] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4483.045019] Lustre: Skipped 1 previous similar message [ 4483.230545] Lustre: server umount lustre-MDT0000 complete [ 4484.485270] LustreError: 296681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773945978 with bad export cookie 17984296436492108719 [ 4484.488336] LustreError: 296681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4484.488484] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4484.602713] Lustre: server umount lustre-MDT0001 complete [ 4488.266846] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4489.472599] Key type lgssc unregistered [ 4489.617547] LNet: 299891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4489.621315] LNetError: 299891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4489.634359] LNet: Removed LNI 192.168.202.116@tcp [ 4489.921110] Key type .llcrypt unregistered [ 4489.923309] Key type ._llcrypt unregistered [ 4498.789701] Key type ._llcrypt registered [ 4498.791537] Key type .llcrypt registered [ 4498.835719] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4499.198028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4499.245230] alg: No test for adler32 (adler32-zlib) [ 4500.097126] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 4500.183959] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4501.767162] Key type lgssc registered [ 4502.135979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4504.637432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4504.645592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.812235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.023229] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4510.002732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.126410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4511.394159] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4513.419929] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4514.817553] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4517.241314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4517.370565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.081999] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4520.923077] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4535.781981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.782837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4535.783868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.789368] Lustre: Skipped 2 previous similar messages [ 4536.818442] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 4536.823091] Lustre: Skipped 2 previous similar messages [ 4539.858834] Lustre: server umount lustre-MDT0000 complete [ 4540.896888] LustreError: 301269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.901018] LustreError: 301269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4541.015370] LustreError: 301248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946034 with bad export cookie 2401627011381741823 [ 4541.019310] LustreError: 301248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4541.019592] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.923825] LustreError: 301293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.937502] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 4546.016106] LustreError: 301265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.016473] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4546.025401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4547.184885] Lustre: server umount lustre-MDT0001 complete [ 4549.264278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4552.088800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4565.471694] LustreError: 300579:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f8feff95 x1860117193580032/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4565.585440] LustreError: 302870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.590218] LustreError: 302870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4565.615335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4565.640469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4566.895187] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4569.789578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.946273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4571.323153] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4573.233760] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4576.102664] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4577.908360] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4579.157247] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4579.749807] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4581.347729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.351209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4581.351270] Lustre: Skipped 1 previous similar message [ 4581.354626] Lustre: Skipped 1 previous similar message [ 4581.356263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4587.234645] Lustre: server umount lustre-OST0000 complete [ 4590.559878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4590.561843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.565561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4590.567104] Lustre: Skipped 2 previous similar messages [ 4594.739313] Lustre: server umount lustre-MDT0000 complete [ 4596.306658] LustreError: 305950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946089 with bad export cookie 2401627011381748977 [ 4596.309891] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4596.311283] LustreError: 305950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4596.446997] Lustre: server umount lustre-MDT0001 complete [ 4600.648933] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4601.884659] Key type lgssc unregistered [ 4602.024520] LNet: 306972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4602.027455] LNetError: 306972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4602.034754] LNet: Removed LNI 192.168.202.116@tcp [ 4602.546442] Key type .llcrypt unregistered [ 4602.548662] Key type ._llcrypt unregistered [ 4611.721101] Key type ._llcrypt registered [ 4611.722080] Key type .llcrypt registered [ 4611.755868] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4612.095413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4612.124697] alg: No test for adler32 (adler32-zlib) [ 4612.969221] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 4613.051310] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4614.631144] Key type lgssc registered [ 4614.988254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4617.546400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4617.552296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4618.708485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.957170] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4622.654346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.779861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4624.043160] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4625.739836] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4626.915521] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4627.917760] LustreError: 308375:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4627.920574] LustreError: 308375:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4627.922773] LustreError: 308375:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 4627.936561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.936629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4627.937050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.940118] Lustre: Skipped 1 previous similar message [ 4633.056715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4633.058668] Lustre: Skipped 2 previous similar messages [ 4638.176812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.179076] Lustre: Skipped 1 previous similar message [ 4642.271185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4642.342482] Lustre: server umount lustre-MDT0000 complete [ 4643.296063] LustreError: 308347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.300778] LustreError: 308347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4643.561858] LustreError: 309735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946137 with bad export cookie 18243761755775143852 [ 4643.565242] LustreError: 309735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4643.566330] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4643.693504] Lustre: server umount lustre-MDT0001 complete [ 4648.378071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4648.469125] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4648.479937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4648.568631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.941919] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4652.750529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.785109] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4652.794521] Lustre: 310708:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4652.892422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4654.239898] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4656.043054] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4657.307078] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4660.004900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4660.099657] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4660.105137] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4660.109983] LustreError: 312364:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 4660.114659] LustreError: 312364:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4660.118214] LustreError: 312364:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4660.149217] Lustre: server umount lustre-OST0000 complete [ 4660.150923] LustreError: 312364:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4662.876363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.946699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4663.012531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.851387] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4664.995937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4666.622145] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4669.168686] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4669.202658] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4669.236118] Lustre: lustre-OST0001: new disk, initializing [ 4669.238333] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4671.228985] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4673.028403] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4673.964166] Lustre: server umount lustre-OST0001 complete [ 4674.562723] LustreError: 312830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.552119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4675.554718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.560824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4675.562574] Lustre: Skipped 1 previous similar message [ 4679.650844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4679.651571] LustreError: 313206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4679.652681] Lustre: Skipped 1 previous similar message [ 4679.660077] LustreError: 313206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4681.290631] Lustre: server umount lustre-OST0000 complete [ 4683.743786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4683.747073] LustreError: Skipped 1 previous similar message [ 4683.749084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4683.754243] Lustre: Skipped 1 previous similar message [ 4688.819677] Lustre: server umount lustre-MDT0000 complete [ 4689.888268] LustreError: 310719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4689.892628] LustreError: 310719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4690.103158] LustreError: 311429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946183 with bad export cookie 18243761755775144384 [ 4690.106312] LustreError: 311429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4690.106854] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4690.222133] Lustre: server umount lustre-MDT0001 complete [ 4693.587578] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4694.624472] Key type lgssc unregistered [ 4694.760426] LNet: 315611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4694.762847] LNetError: 315611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4694.770381] LNet: Removed LNI 192.168.202.116@tcp [ 4695.101643] Key type .llcrypt unregistered [ 4695.102681] Key type ._llcrypt unregistered [ 4726.113974] Key type ._llcrypt registered [ 4726.115046] Key type .llcrypt registered [ 4726.153157] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4732.503701] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4732.939368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4732.948185] alg: No test for adler32 (adler32-zlib) [ 4733.801060] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 4733.880376] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4735.455149] Key type lgssc registered [ 4735.802093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4747.272711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4747.279754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.377728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4748.389165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4748.430067] Lustre: lustre-MDT0000: new disk, initializing [ 4748.453536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4748.459278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4749.746967] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4753.913107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4753.954537] Lustre: 320480:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4753.979019] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4753.980709] Lustre: Skipped 1 previous similar message [ 4754.015441] Lustre: lustre-MDT0001: new disk, initializing [ 4754.047306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4754.058370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4754.061261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4755.432983] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4758.511736] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4759.672673] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4761.957059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4762.084549] Lustre: lustre-OST0000: new disk, initializing [ 4762.086569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4762.118954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.036728] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4765.177417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4765.182820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4765.214101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4767.110570] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4768.463785] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4768.535221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4769.821469] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4769.894381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4775.394957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4775.395159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.397701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4775.400167] Lustre: Skipped 1 previous similar message [ 4775.405987] Lustre: Skipped 1 previous similar message [ 4776.685662] Lustre: server umount lustre-OST0000 complete [ 4780.000219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4780.002626] LustreError: Skipped 1 previous similar message [ 4780.004015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4780.008451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4784.409905] Lustre: server umount lustre-MDT0000 complete [ 4785.547270] LustreError: 322220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946279 with bad export cookie 8467158799265185459 [ 4785.549768] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4785.550792] LustreError: 322220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4785.671044] Lustre: server umount lustre-MDT0001 complete [ 4788.636867] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:51:20 (1773946280) [ 4791.113381] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:51:23 (1773946283) [ 4793.936804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.116059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4795.311165] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4797.986753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.384814] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4801.087278] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4802.385164] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4804.960125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.105459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4805.107587] Lustre: Skipped 1 previous similar message [ 4807.043080] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4809.038006] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4827.967451] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4828.487926] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4828.979700] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4829.557300] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4830.283916] Lustre: Failing over lustre-MDT0000 [ 4830.561743] Lustre: server umount lustre-MDT0000 complete [ 4834.271846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4834.276504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4834.283815] Lustre: Skipped 1 previous similar message [ 4834.286944] LustreError: 324828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.295022] LustreError: 324828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4834.786306] LustreError: 324825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.794436] LustreError: 324825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4839.904108] LustreError: 325539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4839.908676] LustreError: 325539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4843.512269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.561912] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4843.662154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4843.688093] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4845.027159] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4845.111470] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4849.130891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4849.144021] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4849.168456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4850.344068] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4850.933962] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4852.808442] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4853.576691] Lustre: Failing over lustre-MDT0000 [ 4853.784806] Lustre: server umount lustre-MDT0000 complete [ 4854.241252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4854.242121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4854.243178] LustreError: 326983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.254547] Lustre: Skipped 4 previous similar messages [ 4859.361559] LustreError: 324829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.367704] LustreError: 324829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4866.713056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4866.751297] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4866.859752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4866.898424] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4868.157459] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4870.627467] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4872.166798] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4872.168455] Lustre: Skipped 2 previous similar messages [ 4872.175846] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4872.197155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4873.528148] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4874.164399] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4877.284024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4877.286043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4877.287451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4877.287459] Lustre: Skipped 2 previous similar messages [ 4877.299618] Lustre: Skipped 2 previous similar messages [ 4882.401318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4882.404582] Lustre: Skipped 4 previous similar messages [ 4882.871362] Lustre: server umount lustre-MDT0000 complete [ 4884.348584] LustreError: 324811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946377 with bad export cookie 8467158799265188140 [ 4884.351539] LustreError: 324811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4884.351655] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4884.468202] Lustre: server umount lustre-MDT0001 complete [ 4896.359227] Lustre: server umount lustre-OST0000 complete [ 4902.054847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.223118] LustreError: 330625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.228661] LustreError: 330625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4902.248492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4903.594552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4906.432134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.797503] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4909.482455] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4910.550165] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4912.754496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4914.810447] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4916.654751] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4928.230684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4937.696508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4937.699834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.704767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.863456] Lustre: server umount lustre-OST0000 complete [ 4947.284143] Lustre: server umount lustre-MDT0000 complete [ 4948.521363] LustreError: 330607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946442 with bad export cookie 8467158799265188847 [ 4948.527537] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.529309] LustreError: 330607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4948.650604] Lustre: server umount lustre-MDT0001 complete [ 4951.938429] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4953.008592] Key type lgssc unregistered [ 4953.133534] LNet: 333822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4953.135981] LNetError: 333822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4953.143508] LNet: Removed LNI 192.168.202.116@tcp [ 4953.463336] Key type .llcrypt unregistered [ 4953.464497] Key type ._llcrypt unregistered [ 4962.736466] Key type ._llcrypt registered [ 4962.737327] Key type .llcrypt registered [ 4962.771255] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4963.211571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4963.265073] alg: No test for adler32 (adler32-zlib) [ 4964.124655] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 4964.206421] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4965.783160] Key type lgssc registered [ 4966.134443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4968.612618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4968.618172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.765373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.990447] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4973.583669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4973.702630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4974.923711] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4976.519478] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4977.673112] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4979.760285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4979.889417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4980.902118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4981.618575] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4983.261266] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4989.409408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.409586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4989.411703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4989.413896] Lustre: Skipped 1 previous similar message [ 4994.069799] Lustre: server umount lustre-MDT0000 complete [ 4994.528934] LustreError: 337447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.535631] LustreError: 337447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4995.197241] LustreError: 335281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946488 with bad export cookie 10133899918471943806 [ 4995.199865] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4995.200415] LustreError: 335281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4995.311583] Lustre: server umount lustre-MDT0001 complete [ 5006.744834] Lustre: server umount lustre-OST0000 complete [ 5010.942103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.098178] LustreError: 338673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.120593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5012.306738] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5015.101564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.217218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5016.480910] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5018.174270] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5019.344730] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5021.546382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5023.583696] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5024.743761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5025.312639] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5034.977548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5034.979538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5034.980903] Lustre: Skipped 3 previous similar messages [ 5034.982691] Lustre: Skipped 3 previous similar messages [ 5038.169258] Lustre: server umount lustre-MDT0000 complete [ 5039.511380] LustreError: 338653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946533 with bad export cookie 10133899918471944905 [ 5039.514648] LustreError: 338653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5039.516418] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5039.627541] Lustre: server umount lustre-MDT0001 complete [ 5051.222265] Lustre: server umount lustre-OST0000 complete [ 5055.932847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5056.103940] LustreError: 342244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5056.129568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5056.131451] Lustre: Skipped 1 previous similar message [ 5057.403225] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5060.158571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.562548] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5063.242510] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5064.387067] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5066.759688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5069.028717] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5070.823587] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5081.194267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5085.664588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5085.673869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.686746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.691073] Lustre: Skipped 1 previous similar message [ 5086.690679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5088.844334] Lustre: server umount lustre-OST0000 complete [ 5091.296485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5091.301191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.308375] Lustre: Skipped 1 previous similar message [ 5091.311733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.317144] Lustre: Skipped 2 previous similar messages [ 5096.542785] Lustre: server umount lustre-MDT0000 complete [ 5096.928099] LustreError: 342245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.932116] LustreError: 342245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5097.719270] LustreError: 342226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946591 with bad export cookie 10133899918471945913 [ 5097.721672] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.723064] LustreError: 342226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5097.839526] Lustre: server umount lustre-MDT0001 complete [ 5101.091039] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5101.977790] Key type lgssc unregistered [ 5102.101364] LNet: 345440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5102.103500] LNetError: 345440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5102.109316] LNet: Removed LNI 192.168.202.116@tcp [ 5102.393975] Key type .llcrypt unregistered [ 5102.394953] Key type ._llcrypt unregistered [ 5111.252087] Key type ._llcrypt registered [ 5111.253139] Key type .llcrypt registered [ 5111.297564] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5111.604914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5111.641216] alg: No test for adler32 (adler32-zlib) [ 5112.485697] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5112.565440] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5114.143146] Key type lgssc registered [ 5114.493383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5117.112462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5117.117918] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.121405] LustreError: 346840:0:(osd_handler.c:8531:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5117.130102] LustreError: 346840:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5117.133478] LustreError: 346840:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5117.136937] LustreError: 346840:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5117.140864] LustreError: 346840:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5119.418859] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5119.421609] LustreError: 347237:0:(osd_handler.c:8531:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5119.429976] LustreError: 347237:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5119.433239] LustreError: 347237:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5119.435785] LustreError: 347237:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5119.439834] LustreError: 347237:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5125.274686] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5126.229996] Key type lgssc unregistered [ 5126.351356] LNet: 348220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.353400] LNetError: 348220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5126.360266] LNet: Removed LNI 192.168.202.116@tcp [ 5126.655369] Key type .llcrypt unregistered [ 5126.656289] Key type ._llcrypt unregistered [ 5132.136040] Key type ._llcrypt registered [ 5132.136979] Key type .llcrypt registered [ 5132.175953] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5138.251702] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5138.560670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5138.590373] alg: No test for adler32 (adler32-zlib) [ 5139.430800] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5139.504910] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5141.087209] Key type lgssc registered [ 5141.468167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5154.387137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5154.393774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.482865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5155.493243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5155.529734] Lustre: lustre-MDT0000: new disk, initializing [ 5155.554086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5155.559920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5157.093259] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5161.417692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.455123] Lustre: 351314:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5161.475860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5161.477763] Lustre: Skipped 1 previous similar message [ 5161.517308] Lustre: lustre-MDT0001: new disk, initializing [ 5161.548389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5161.561304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5161.565314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5162.751621] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5165.726849] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5166.831227] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5168.999274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5169.097446] Lustre: lustre-OST0000: new disk, initializing [ 5169.099403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5169.127764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5170.914584] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5173.866723] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5174.775207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5174.778116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5174.817027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5175.125387] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5175.194306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5176.397938] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5176.463487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5179.874487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5179.874562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5179.878363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5179.879702] LustreError: Skipped 1 previous similar message [ 5183.187573] Lustre: server umount lustre-OST0000 complete [ 5184.992404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.992841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5184.995658] Lustre: Skipped 2 previous similar messages [ 5184.997234] Lustre: Skipped 1 previous similar message [ 5190.112227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5190.113777] Lustre: Skipped 2 previous similar messages [ 5190.490798] Lustre: server umount lustre-MDT0000 complete [ 5191.658178] LustreError: 353056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946685 with bad export cookie 16759580938047967271 [ 5191.661258] LustreError: 353056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5191.662196] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5191.776859] Lustre: server umount lustre-MDT0001 complete [ 5194.335097] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5194.856406] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:58:07 (1773946687) [ 5196.917713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.087498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5198.339463] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5201.047143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.165267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5202.396487] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5204.107497] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5205.246690] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5207.447032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5207.521040] LustreError: 356958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.525861] LustreError: 356958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5209.348351] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5211.194713] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5213.360752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5213.418358] Lustre: lustre-OST0001: new disk, initializing [ 5213.419959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5213.450241] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5213.453736] Lustre: Skipped 1 previous similar message [ 5215.411453] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5218.735649] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5220.355211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5220.358089] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5220.387953] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5224.937678] Lustre: lustre-OST0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 5225.442722] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5225.443809] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.447053] LustreError: Skipped 1 previous similar message [ 5226.685249] Lustre: server umount lustre-OST0001 complete [ 5230.565117] LustreError: 356958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.572025] LustreError: 356958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5235.687480] LustreError: 356960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.692618] LustreError: 356960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5240.803085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.805725] LustreError: 356958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.807044] Lustre: Skipped 2 previous similar messages [ 5240.812797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5240.814717] Lustre: Skipped 3 previous similar messages [ 5245.294411] Lustre: server umount lustre-OST0000 complete [ 5247.455928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5247.458589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.041386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5251.046395] Lustre: Skipped 2 previous similar messages [ 5253.224690] Lustre: server umount lustre-MDT0000 complete [ 5254.618346] LustreError: 355347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946748 with bad export cookie 16759580938047968139 [ 5254.621016] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5254.621420] LustreError: 355347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5260.851671] Lustre: server umount lustre-MDT0001 complete [ 5265.315342] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5266.815570] Key type lgssc unregistered [ 5267.007656] LNet: 359717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5267.012459] LNetError: 359717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5267.020663] LNet: Removed LNI 192.168.202.116@tcp [ 5267.526485] Key type .llcrypt unregistered [ 5267.528458] Key type ._llcrypt unregistered [ 5290.047683] Key type ._llcrypt registered [ 5290.050299] Key type .llcrypt registered [ 5290.124777] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5290.673670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5290.707548] alg: No test for adler32 (adler32-zlib) [ 5291.590503] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5291.706098] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5293.303098] Key type lgssc registered [ 5293.710305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5296.750455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5296.761066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.875867] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5297.884296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5297.944252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5299.541830] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5302.991757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.046096] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5303.062308] Lustre: 362473:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5303.195148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5304.934987] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5307.521247] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5309.103869] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5313.504549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5313.509914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5313.519127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5314.018671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.027168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5314.030696] Lustre: Skipped 1 previous similar message [ 5316.196068] Lustre: server umount lustre-MDT0000 complete [ 5317.842290] LustreError: 362466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946811 with bad export cookie 15354091834065035269 [ 5317.848804] LustreError: 362466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5317.848939] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5318.006215] Lustre: server umount lustre-MDT0001 complete [ 5321.918885] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:00:14 (1773946814) [ 5325.006530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.208695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.928063] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5330.401713] LustreError: 364766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.405728] LustreError: 364766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5330.626924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5330.736323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5331.934466] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5334.014256] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5335.175196] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5338.322407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5338.406881] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5338.469109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5340.528718] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5342.839787] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5346.934403] Lustre: Permanently deactivating lustre-OST0000 [ 5346.943564] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5347.040958] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5347.718730] LustreError: 367217:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5347.725395] LustreError: 367217:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5347.730062] LustreError: 367217:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5348.361079] LustreError: 367265:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5348.363033] LustreError: 367265:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5355.353235] Lustre: server umount lustre-OST0000 complete [ 5356.448952] LustreError: 367514:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5356.451629] LustreError: 367514:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5359.585576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.586328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5359.591451] Lustre: Skipped 1 previous similar message [ 5359.596242] Lustre: Skipped 1 previous similar message [ 5364.705206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5364.708857] Lustre: Skipped 1 previous similar message [ 5366.790738] Lustre: server umount lustre-MDT0000 complete [ 5368.147262] LustreError: 364748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946861 with bad export cookie 15354091834065035836 [ 5368.152424] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5368.157981] LustreError: 364748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5377.557033] Lustre: server umount lustre-MDT0001 complete [ 5381.716969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.234199] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5384.380591] LustreError: 368666:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5387.246222] Lustre: 369005:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.116@tcp [ 5388.266511] Lustre: server umount MGS complete [ 5393.032898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.207951] LustreError: 369728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5393.212568] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5393.215576] Lustre: Skipped 1 previous similar message [ 5393.242800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.683422] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5397.808704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.901914] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5399.135449] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5400.914722] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5402.021417] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5402.627719] Lustre: Permanently reactivating lustre-OST0000 [ 5402.634774] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5402.636617] Lustre: Skipped 2 previous similar messages [ 5429.727256] LustreError: 371367:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5434.006113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5434.180853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5434.184729] Lustre: Skipped 1 previous similar message [ 5436.406777] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5438.975942] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5448.672677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5448.678223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.689936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5449.697605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5452.184589] Lustre: server umount lustre-OST0000 complete [ 5454.304433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5454.309115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.316331] Lustre: Skipped 1 previous similar message [ 5459.759928] Lustre: server umount lustre-MDT0000 complete [ 5459.936168] LustreError: 370438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.942885] LustreError: 370438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5461.536256] LustreError: 371074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773946955 with bad export cookie 15354091834065036844 [ 5461.542559] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.543803] LustreError: 371074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5461.676148] Lustre: server umount lustre-MDT0001 complete [ 5466.029660] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5467.184658] Key type lgssc unregistered [ 5467.318455] LNet: 374073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5467.320910] LNetError: 374073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5467.329376] LNet: Removed LNI 192.168.202.116@tcp [ 5467.657697] Key type .llcrypt unregistered [ 5467.659844] Key type ._llcrypt unregistered [ 5473.728423] Key type ._llcrypt registered [ 5473.729337] Key type .llcrypt registered [ 5473.762854] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5479.442554] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5479.810895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5479.838595] alg: No test for adler32 (adler32-zlib) [ 5480.677170] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5480.750400] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5482.327184] Key type lgssc registered [ 5482.842707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5494.272569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5494.279793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.365769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5495.375377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5495.433053] Lustre: lustre-MDT0000: new disk, initializing [ 5495.455358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.461072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5496.649529] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5500.590145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.632267] Lustre: 377169:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5500.653634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5500.656138] Lustre: Skipped 1 previous similar message [ 5500.690373] Lustre: lustre-MDT0001: new disk, initializing [ 5500.721401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5500.732654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5500.736155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5501.920508] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5504.928150] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5506.190202] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5508.675870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5508.820482] Lustre: lustre-OST0000: new disk, initializing [ 5508.822731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5508.857273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.319879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5509.322729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5509.362855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5510.673434] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5513.804439] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5515.125353] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5515.197914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5516.575201] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5516.646818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5519.841423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5519.842984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.844396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5519.847281] Lustre: Skipped 1 previous similar message [ 5523.348422] Lustre: server umount lustre-OST0000 complete [ 5524.960938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.961928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5524.964379] Lustre: Skipped 1 previous similar message [ 5524.966286] Lustre: Skipped 1 previous similar message [ 5530.080320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.082533] Lustre: Skipped 2 previous similar messages [ 5530.788726] Lustre: server umount lustre-MDT0000 complete [ 5532.057296] LustreError: 377159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773947025 with bad export cookie 12508942145191272117 [ 5532.060986] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.061380] LustreError: 377159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5532.186099] Lustre: server umount lustre-MDT0001 complete [ 5534.825250] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5535.300648] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:03:47 (1773947027) [ 5537.463599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.636308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.816719] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5541.499322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.655524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5542.807368] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5544.632287] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5545.754112] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5547.911318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5549.994368] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5552.383615] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5553.124631] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5554.626940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5556.191711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5556.193663] LustreError: Skipped 1 previous similar message [ 5556.194780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5556.198197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5557.216663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5561.867525] Lustre: server umount lustre-OST0000 complete [ 5567.458533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.458778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5567.459733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5567.459741] Lustre: Skipped 2 previous similar messages [ 5567.465750] Lustre: Skipped 2 previous similar messages [ 5569.501527] Lustre: server umount lustre-MDT0000 complete [ 5571.013273] LustreError: 381197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773947064 with bad export cookie 12508942145191272985 [ 5571.018474] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5571.023068] LustreError: 381197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5571.194840] Lustre: server umount lustre-MDT0001 complete [ 5575.096138] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5576.381291] Key type lgssc unregistered [ 5576.500484] LNet: 384586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5576.504367] LNetError: 384586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5576.517468] LNet: Removed LNI 192.168.202.116@tcp [ 5576.918487] Key type .llcrypt unregistered [ 5576.920789] Key type ._llcrypt unregistered [ 5585.630688] Key type ._llcrypt registered [ 5585.632276] Key type .llcrypt registered [ 5585.676805] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5585.812490] Key type .llcrypt unregistered [ 5585.814582] Key type ._llcrypt unregistered [ 5592.939905] Key type ._llcrypt registered [ 5592.940957] Key type .llcrypt registered [ 5592.973709] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5593.378951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5593.385231] alg: No test for adler32 (adler32-zlib) [ 5594.232729] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5594.310965] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5595.887111] Key type lgssc registered [ 5596.233120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5598.941550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5598.950046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.118574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.526579] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5605.381344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5605.535687] LustreError: 386736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.540407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5605.542932] LustreError: 386736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5608.027710] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5609.634580] LustreError: 386740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5609.644714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5609.649096] LustreError: 386740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5610.720056] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5613.845340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.992577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5615.267720] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5624.290905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.291210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5624.296521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5624.300370] Lustre: Skipped 1 previous similar message [ 5628.424311] Lustre: server umount lustre-OST0000 complete [ 5632.993173] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5632.995256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.000168] Lustre: Skipped 1 previous similar message [ 5633.002347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.528129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.528647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.669525] Lustre: server umount lustre-MDT0000 complete [ 5637.080946] LustreError: 386719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773947130 with bad export cookie 9330360837739928299 [ 5637.086506] LustreError: 386719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5637.088206] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5637.223195] Lustre: server umount lustre-MDT0001 complete [ 5641.131075] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5642.288483] Key type lgssc unregistered [ 5642.430623] LNet: 389726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5642.435410] LNetError: 389726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5642.445521] LNet: Removed LNI 192.168.202.116@tcp [ 5642.889932] Key type .llcrypt unregistered [ 5642.892251] Key type ._llcrypt unregistered [ 5653.032770] Key type ._llcrypt registered [ 5653.033667] Key type .llcrypt registered [ 5653.067176] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5653.335314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5653.354777] alg: No test for adler32 (adler32-zlib) [ 5654.211104] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5654.290852] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5655.879180] Key type lgssc registered [ 5656.399949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5659.838506] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5659.850482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5675.295365] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5680.415203] LustreError: 391076:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 5707.039230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.091298] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5711.408114] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5714.398545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.643673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.673506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5716.285442] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5720.033979] LustreError: 391102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.039881] LustreError: 391102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5720.384671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5720.579882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5722.137191] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5724.322532] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5726.141368] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5730.271779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5730.273763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.277200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5730.784366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5730.786343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.790095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.810419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.812126] Lustre: Skipped 1 previous similar message [ 5734.921077] Lustre: server umount lustre-OST0000 complete [ 5736.928212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.928814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5736.931398] Lustre: Skipped 1 previous similar message [ 5742.048576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5742.052691] Lustre: Skipped 2 previous similar messages [ 5742.248139] Lustre: server umount lustre-MDT0000 complete [ 5743.981032] LustreError: 393269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773947237 with bad export cookie 11876150427166883717 [ 5743.986555] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5743.987499] LustreError: 393269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5744.132744] Lustre: server umount lustre-MDT0001 complete [ 5748.651795] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5749.921599] Key type lgssc unregistered [ 5750.070626] LNet: 394291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5750.074903] LNetError: 394291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5750.082565] LNet: Removed LNI 192.168.202.116@tcp [ 5750.496331] Key type .llcrypt unregistered [ 5750.498271] Key type ._llcrypt unregistered [ 5761.453958] Key type ._llcrypt registered [ 5761.455422] Key type .llcrypt registered [ 5761.490406] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5761.868946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5761.879380] alg: No test for adler32 (adler32-zlib) [ 5762.726873] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5762.803378] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5764.383109] Key type lgssc registered [ 5764.754453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5767.685643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5767.691389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.843338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.121366] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5773.773440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.918911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5775.363024] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5777.091613] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5778.822825] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5781.970846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.109438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.078270] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5785.735935] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5794.413304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5799.392133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5799.395247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5799.399568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5799.907873] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5799.909623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5799.916160] Lustre: Skipped 1 previous similar message [ 5801.458786] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 5801.464042] Lustre: Skipped 1 previous similar message [ 5803.698206] Lustre: server umount lustre-MDT0000 complete [ 5804.436134] LustreError: 395649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.16@tcp arrived at 1773947298 with bad export cookie 15438448732949405683 [ 5804.439458] LustreError: 395649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5805.026823] LustreError: 395666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.032888] LustreError: 395666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5805.059282] LustreError: 395648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773947298 with bad export cookie 15438448732949405032 [ 5805.063811] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5805.064832] LustreError: 395648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5810.145348] LustreError: 395667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.145807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5810.160739] LustreError: 395667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5810.165514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5810.166770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5811.176499] Lustre: server umount lustre-OST0000 complete [ 5813.958734] Lustre: server umount lustre-MDT0001 complete [ 5817.934273] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5819.444203] Key type lgssc unregistered [ 5819.572626] LNet: 399016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5819.576499] LNetError: 399016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5819.586590] LNet: Removed LNI 192.168.202.116@tcp [ 5820.024983] Key type .llcrypt unregistered [ 5820.027727] Key type ._llcrypt unregistered [ 5831.083775] Key type ._llcrypt registered [ 5831.086160] Key type .llcrypt registered [ 5831.145718] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5831.452630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5831.487040] alg: No test for adler32 (adler32-zlib) [ 5832.334807] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5832.419960] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5834.007173] Key type lgssc registered [ 5834.344668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5837.825355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5837.836624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.038119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.664995] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5844.771662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.929386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5846.631585] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5849.056927] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5850.759652] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5853.934183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5854.064247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.247431] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5858.647558] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5864.300542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5874.160102] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 5874.660085] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5874.663172] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.666379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5874.673218] Lustre: Skipped 1 previous similar message [ 5879.269166] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 5879.273816] Lustre: Skipped 1 previous similar message [ 5884.383146] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5884.392798] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 5884.396081] Lustre: Skipped 2 previous similar messages [ 5884.455544] Lustre: server umount lustre-MDT0001 complete [ 5884.897073] LustreError: 400393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.900995] LustreError: 400393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5890.018435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5890.019112] LustreError: 401539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5890.019503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5890.020905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5892.713207] Lustre: server umount lustre-OST0000 complete [ 5894.221924] Lustre: server umount lustre-MDT0000 complete [ 5899.877014] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5901.107458] Key type lgssc unregistered [ 5901.242484] LNet: 403744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5901.246423] LNetError: 403744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5901.258319] LNet: Removed LNI 192.168.202.116@tcp [ 5901.532765] Key type .llcrypt unregistered [ 5901.533880] Key type ._llcrypt unregistered [ 5911.267995] Key type ._llcrypt registered [ 5911.269550] Key type .llcrypt registered [ 5911.319756] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5918.639411] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5918.980487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5919.006673] alg: No test for adler32 (adler32-zlib) [ 5919.860651] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 5919.941767] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5921.519170] Key type lgssc registered [ 5921.825116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5933.717924] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5939.711584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5939.717926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.809478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5940.821173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5940.862290] Lustre: lustre-MDT0000: new disk, initializing [ 5940.886177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.892888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5942.263680] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5949.699561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5949.860890] Lustre: lustre-OST0000: new disk, initializing [ 5949.863825] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5949.865477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5949.871994] Lustre: Skipped 1 previous similar message [ 5949.901818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.249852] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5955.067144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5955.071347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5955.082691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5956.154894] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5963.813337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.877389] Lustre: 407680:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5963.918302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5963.956864] Lustre: lustre-MDT0001: new disk, initializing [ 5964.003192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5964.017449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5964.020649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5965.196943] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5969.313529] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5969.410742] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5971.035104] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5971.103491] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5974.504306] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 5974.670434] Lustre: server umount lustre-MDT0001 complete [ 5976.548954] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.549675] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5976.550485] LustreError: 409901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.550496] LustreError: 409901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5976.558853] Lustre: Skipped 1 previous similar message [ 5981.665500] LustreError: 409195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.669210] LustreError: 409195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5986.785446] LustreError: 407686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5986.791279] LustreError: 407686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5990.880057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5990.883815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.888881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5991.905180] LustreError: 410256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.024303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5997.024800] LustreError: 407692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.028755] Lustre: Skipped 1 previous similar message [ 6001.119188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6001.178982] Lustre: server umount lustre-OST0000 complete [ 6008.558100] Lustre: server umount lustre-MDT0000 complete [ 6012.574616] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6013.575119] Key type lgssc unregistered [ 6013.695339] LNet: 412252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6013.698906] LNetError: 412252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6013.705448] LNet: Removed LNI 192.168.202.116@tcp [ 6014.046283] Key type .llcrypt unregistered [ 6014.047815] Key type ._llcrypt unregistered [ 6026.380881] Key type ._llcrypt registered [ 6026.382385] Key type .llcrypt registered [ 6026.426391] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6026.546601] Key type .llcrypt unregistered [ 6026.547982] Key type ._llcrypt unregistered [ 6032.570457] Key type ._llcrypt registered [ 6032.571420] Key type .llcrypt registered [ 6032.610228] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6033.096713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6033.106595] alg: No test for adler32 (adler32-zlib) [ 6033.959523] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 6034.049201] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6035.631240] Key type lgssc registered [ 6036.160045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6045.460348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6045.469863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.574457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6046.585022] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6046.621175] Lustre: lustre-MDT0000: new disk, initializing [ 6046.646105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.652527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6048.011163] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6052.333514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6052.381492] Lustre: 415568:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6052.407239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6052.408947] Lustre: Skipped 1 previous similar message [ 6052.451347] Lustre: lustre-MDT0001: new disk, initializing [ 6052.489108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6052.502470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6052.506548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6053.923263] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6057.091748] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6058.381973] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6061.027754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6061.174873] Lustre: lustre-OST0000: new disk, initializing [ 6061.179355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6061.219565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.324674] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6066.570308] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6068.230141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6068.236569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6068.285732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6078.434928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6078.437877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.440360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6078.447101] Lustre: Skipped 1 previous similar message [ 6080.053321] Lustre: server umount lustre-MDT0000 complete [ 6081.619409] LustreError: 416348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773947575 with bad export cookie 3932444861602115803 [ 6081.623366] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6081.746569] Lustre: server umount lustre-MDT0001 complete [ 6093.286672] Lustre: server umount lustre-OST0000 complete [ 6096.702253] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:13:08 (1773947588) [ 6099.482338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.688498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.020902] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6103.940229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.071829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6105.407836] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6107.150726] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6108.309836] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6121.631332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6121.746928] Lustre: Found index 0 for lustre-OST0000, updating log [ 6121.845049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.873400] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6124.908106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6125.675512] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6128.869091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6128.869332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.871494] Lustre: Skipped 2 previous similar messages [ 6128.875958] Lustre: Skipped 1 previous similar message [ 6130.675269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6130.677068] Lustre: Skipped 1 previous similar message [ 6132.770115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6132.772013] Lustre: Skipped 2 previous similar messages [ 6132.958925] Lustre: server umount lustre-OST0000 complete [ 6145.617538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.619529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6145.621248] Lustre: Skipped 1 previous similar message [ 6151.797435] Lustre: server umount lustre-MDT0000 complete [ 6153.230994] LustreError: 420837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773947646 with bad export cookie 3932444861602117189 [ 6153.237746] LustreError: 420837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6153.237812] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6153.381447] Lustre: server umount lustre-MDT0001 complete [ 6156.354586] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:14:08 (1773947648) [ 6183.583383] LNet: 1 local NIs in recovery (showing 1): 192.168.202.116@tcp [ 6740.880558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.099208] LustreError: 425290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.105880] LustreError: 425290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6741.132316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.670974] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6746.298601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.722430] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6749.925153] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6751.282742] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6764.890691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6764.980021] Lustre: Found index 0 for lustre-OST0000, updating log [ 6767.487996] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6768.173663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6769.666237] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6771.683234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.684770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6771.694895] Lustre: Skipped 4 previous similar messages [ 6776.824425] Lustre: server umount lustre-OST0000 complete [ 6778.215244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.216127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6778.222215] Lustre: Skipped 1 previous similar message [ 6780.256107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6780.258099] Lustre: Skipped 4 previous similar messages [ 6784.353049] LustreError: 425291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.364044] LustreError: 425291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6784.377343] Lustre: server umount lustre-MDT0000 complete [ 6785.376305] LustreError: 425309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6785.382157] LustreError: 425309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6786.148267] LustreError: 425269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773948279 with bad export cookie 3932444861602118246 [ 6786.155197] LustreError: 425269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6786.155293] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6786.287910] Lustre: server umount lustre-MDT0001 complete [ 6797.983442] LNet: 1 local NIs in recovery (showing 1): 192.168.202.116@tcp [ 6805.337366] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6810.054794] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6826.062733] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:25:18 (1773948318) [ 6833.707258] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6835.139782] Key type lgssc unregistered [ 6835.305963] LNet: 431556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6835.310723] LNetError: 431556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6894.693779] LNet: Removed LNI 192.168.202.116@tcp [ 6895.210828] Key type .llcrypt unregistered [ 6895.213164] Key type ._llcrypt unregistered [ 6902.787602] Key type ._llcrypt registered [ 6902.788743] Key type .llcrypt registered [ 6902.844678] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6903.297448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6903.343729] alg: No test for adler32 (adler32-zlib) [ 6904.211965] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 6904.294749] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6905.871155] Key type lgssc registered [ 6906.475240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6908.492828] LNet: Added LNI 192.168.202.116@tcp100 [8/256/0/180] [ 6908.930902] LNet: Added LNI 192.168.202.116@tcp101 [8/256/0/180] [ 6909.368169] LNet: Added LNI 192.168.202.116@tcp102 [8/256/0/180] [ 6909.805948] LNet: Added LNI 192.168.202.116@tcp103 [8/256/0/180] [ 6910.245208] LNet: Added LNI 192.168.202.116@tcp104 [8/256/0/180] [ 6910.701819] LNet: Added LNI 192.168.202.116@tcp105 [8/256/0/180] [ 6911.138699] LNet: Added LNI 192.168.202.116@tcp106 [8/256/0/180] [ 6911.550371] LNet: Added LNI 192.168.202.116@tcp107 [8/256/0/180] [ 6911.970518] LNet: Added LNI 192.168.202.116@tcp108 [8/256/0/180] [ 6912.372919] LNet: Added LNI 192.168.202.116@tcp109 [8/256/0/180] [ 6912.693654] LNet: Added LNI 192.168.202.116@tcp110 [8/256/0/180] [ 6913.025098] LNet: Added LNI 192.168.202.116@tcp111 [8/256/0/180] [ 6913.369914] LNet: Added LNI 192.168.202.116@tcp112 [8/256/0/180] [ 6913.648129] LNet: Added LNI 192.168.202.116@tcp113 [8/256/0/180] [ 6913.915709] LNet: Added LNI 192.168.202.116@tcp114 [8/256/0/180] [ 6914.280719] LNet: Added LNI 192.168.202.116@tcp115 [8/256/0/180] [ 6914.660536] LNet: Added LNI 192.168.202.116@tcp116 [8/256/0/180] [ 6914.966747] LNet: Added LNI 192.168.202.116@tcp117 [8/256/0/180] [ 6915.362319] LNet: Added LNI 192.168.202.116@tcp118 [8/256/0/180] [ 6915.802965] LNet: Added LNI 192.168.202.116@tcp119 [8/256/0/180] [ 6916.192853] LNet: Added LNI 192.168.202.116@tcp120 [8/256/0/180] [ 6916.612027] LNet: Added LNI 192.168.202.116@tcp121 [8/256/0/180] [ 6917.001319] LNet: Added LNI 192.168.202.116@tcp122 [8/256/0/180] [ 6917.325826] LNet: Added LNI 192.168.202.116@tcp123 [8/256/0/180] [ 6917.607860] LNet: Added LNI 192.168.202.116@tcp124 [8/256/0/180] [ 6917.877656] LNet: Added LNI 192.168.202.116@tcp125 [8/256/0/180] [ 6918.138473] LNet: Added LNI 192.168.202.116@tcp126 [8/256/0/180] [ 6918.401980] LNet: Added LNI 192.168.202.116@tcp127 [8/256/0/180] [ 6918.667505] LNet: Added LNI 192.168.202.116@tcp128 [8/256/0/180] [ 6918.985038] LNet: Added LNI 192.168.202.116@tcp129 [8/256/0/180] [ 6919.393794] LNet: Added LNI 192.168.202.116@tcp130 [8/256/0/180] [ 6919.698420] LNet: Added LNI 192.168.202.116@tcp131 [8/256/0/180] [ 6920.004561] LNet: Added LNI 192.168.202.116@tcp132 [8/256/0/180] [ 6920.359162] LNet: Added LNI 192.168.202.116@tcp133 [8/256/0/180] [ 6920.745163] LNet: Added LNI 192.168.202.116@tcp134 [8/256/0/180] [ 6921.118812] LNet: Added LNI 192.168.202.116@tcp135 [8/256/0/180] [ 6923.555261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6923.560167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.603519] Lustre: 434535:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 6924.674429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6924.686347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6924.724863] Lustre: lustre-MDT0000: new disk, initializing [ 6924.754609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6924.762774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6925.667445] LNetError: 434557:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 6925.670912] LNetError: Protocol error connecting to 192.168.202.116@tcp115 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 6926.117249] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6930.101500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.178143] Lustre: 434552:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6930.219402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6930.222654] Lustre: Skipped 1 previous similar message [ 6930.267471] Lustre: lustre-MDT0001: new disk, initializing [ 6930.280571] Lustre: 435336:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 6930.283047] Lustre: 435336:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6930.321155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6930.332859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6930.336617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6931.314234] Lustre: 435522:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 6931.317038] Lustre: 435522:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6931.525228] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6934.452552] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6935.956155] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6938.335076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6938.437602] Lustre: lustre-OST0000: new disk, initializing [ 6938.439349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6938.467305] Lustre: 436303:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.116@tcp: can't add NID 192.168.202.116@tcp130: rc = -75 [ 6938.469643] Lustre: 436303:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6938.484687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6940.420476] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6940.697907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6940.704783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6940.779986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6943.982185] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6950.880396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6950.885567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.895465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6951.396109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6951.396201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6951.400639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.250859] Lustre: server umount lustre-OST0000 complete [ 6956.000806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6956.006482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.015436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.021634] Lustre: Skipped 1 previous similar message [ 6960.818430] Lustre: server umount lustre-MDT0000 complete [ 6962.662158] LustreError: 434543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773948456 with bad export cookie 12611238495588867909 [ 6962.669381] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6962.670376] LustreError: 434543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6962.827489] Lustre: server umount lustre-MDT0001 complete [ 6965.728914] LNetError: 432001:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.116 [ 6965.735057] LNetError: Protocol error connecting to 192.168.202.116@tcp115 at host 192.168.202.116:988: is it running a compatible version of Lustre? [ 6967.157052] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6968.549644] Key type lgssc unregistered [ 6968.713530] LNet: 438079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6968.717842] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6968.721884] LNet: Removed LNI 192.168.202.116@tcp [ 6969.759189] LNet: Removed LNI 192.168.202.116@tcp100 [ 6969.761752] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6969.766257] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6969.770930] LNet: Removed LNI 192.168.202.116@tcp101 [ 6970.783201] LNet: Removed LNI 192.168.202.116@tcp102 [ 6970.785792] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6970.790054] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6970.794904] LNet: Removed LNI 192.168.202.116@tcp103 [ 6971.807175] LNet: Removed LNI 192.168.202.116@tcp104 [ 6972.831186] LNet: Removed LNI 192.168.202.116@tcp105 [ 6972.832808] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6972.834985] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6972.837478] LNet: Removed LNI 192.168.202.116@tcp106 [ 6973.855198] LNet: Removed LNI 192.168.202.116@tcp107 [ 6974.879182] LNet: Removed LNI 192.168.202.116@tcp108 [ 6975.903221] LNet: Removed LNI 192.168.202.116@tcp109 [ 6976.927215] LNet: Removed LNI 192.168.202.116@tcp110 [ 6976.929974] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6976.934771] LNetError: 438079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6976.940059] LNet: Removed LNI 192.168.202.116@tcp111 [ 6977.951134] LNet: Removed LNI 192.168.202.116@tcp112 [ 6978.975142] LNet: Removed LNI 192.168.202.116@tcp113 [ 6979.999199] LNet: Removed LNI 192.168.202.116@tcp114 [ 6981.023167] LNet: Removed LNI 192.168.202.116@tcp115 [ 6982.047123] LNet: Removed LNI 192.168.202.116@tcp116 [ 6982.049323] LNet: Removed LNI 192.168.202.116@tcp117 [ 6982.051435] LNet: Removed LNI 192.168.202.116@tcp118 [ 6982.053354] LNet: Removed LNI 192.168.202.116@tcp119 [ 6982.055435] LNet: Removed LNI 192.168.202.116@tcp120 [ 6982.057539] LNet: Removed LNI 192.168.202.116@tcp121 [ 6982.059557] LNet: Removed LNI 192.168.202.116@tcp122 [ 6982.061664] LNet: Removed LNI 192.168.202.116@tcp123 [ 6982.063735] LNet: Removed LNI 192.168.202.116@tcp124 [ 6982.065810] LNet: Removed LNI 192.168.202.116@tcp125 [ 6982.067841] LNet: Removed LNI 192.168.202.116@tcp126 [ 6982.069880] LNet: Removed LNI 192.168.202.116@tcp127 [ 6982.071966] LNet: Removed LNI 192.168.202.116@tcp128 [ 6982.074055] LNet: Removed LNI 192.168.202.116@tcp129 [ 6982.075857] LNet: Removed LNI 192.168.202.116@tcp130 [ 6982.077991] LNet: Removed LNI 192.168.202.116@tcp131 [ 6982.080165] LNet: Removed LNI 192.168.202.116@tcp132 [ 6982.082318] LNet: Removed LNI 192.168.202.116@tcp133 [ 6982.084437] LNet: Removed LNI 192.168.202.116@tcp134 [ 6982.092517] LNet: Removed LNI 192.168.202.116@tcp135 [ 6982.349523] Key type .llcrypt unregistered [ 6982.350503] Key type ._llcrypt unregistered [ 6992.118499] Key type ._llcrypt registered [ 6992.119787] Key type .llcrypt registered [ 6992.174795] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6999.342405] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6999.628096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6999.660317] alg: No test for adler32 (adler32-zlib) [ 7000.504893] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 7000.580098] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7002.159144] Key type lgssc registered [ 7002.733805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7019.081323] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:28:31 (1773948511) [ 7027.250509] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7028.598579] Key type lgssc unregistered [ 7028.745484] LNet: 442086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7028.749380] LNetError: 442086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7028.762467] LNet: Removed LNI 192.168.202.116@tcp [ 7029.175759] Key type .llcrypt unregistered [ 7029.177909] Key type ._llcrypt unregistered [ 7036.788018] Key type ._llcrypt registered [ 7036.789242] Key type .llcrypt registered [ 7036.824114] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7037.125269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7037.142821] alg: No test for adler32 (adler32-zlib) [ 7037.982768] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 7038.061346] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7039.639156] Key type lgssc registered [ 7040.106849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7044.276382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7044.286781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.405631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7045.424227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7045.481554] Lustre: lustre-MDT0000: new disk, initializing [ 7045.508431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7045.514571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7047.223389] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7052.706597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.777687] Lustre: 443260:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7052.820949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7052.824383] Lustre: Skipped 1 previous similar message [ 7052.871357] Lustre: lustre-MDT0001: new disk, initializing [ 7052.919055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7052.939190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7052.945736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7054.569670] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7058.725821] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7060.437830] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7063.685220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7063.785443] Lustre: lustre-OST0000: new disk, initializing [ 7063.787521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7063.813328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7064.032613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7064.038953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7064.096189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7066.292977] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7069.979366] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7071.510340] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing lnet_if_list [ 7073.480291] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7073.496557] LNet: Added LNI 192.168.202.116@tcp10 [8/256/0/180] [ 7075.477362] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7075.489232] LNet: Added LNI 192.168.202.116@tcp11 [8/256/0/180] [ 7077.174337] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7077.193333] LNet: Added LNI 192.168.202.116@tcp12 [8/256/0/180] [ 7078.914264] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7078.932128] LNet: Added LNI 192.168.202.116@tcp13 [8/256/0/180] [ 7080.717589] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7080.737145] LNet: Added LNI 192.168.202.116@tcp14 [8/256/0/180] [ 7082.544527] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7082.562933] LNet: Added LNI 192.168.202.116@tcp15 [8/256/0/180] [ 7084.522898] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7084.541472] LNet: Added LNI 192.168.202.116@tcp16 [8/256/0/180] [ 7086.277234] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7086.289099] LNet: Added LNI 192.168.202.116@tcp17 [8/256/0/180] [ 7088.012884] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7088.025357] LNet: Added LNI 192.168.202.116@tcp18 [8/256/0/180] [ 7089.934282] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7089.949649] LNet: Added LNI 192.168.202.116@tcp19 [8/256/0/180] [ 7091.801708] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7091.814100] LNet: Added LNI 192.168.202.116@tcp20 [8/256/0/180] [ 7093.443912] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7093.452333] LNet: Added LNI 192.168.202.116@tcp21 [8/256/0/180] [ 7095.106829] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7095.116693] LNet: Added LNI 192.168.202.116@tcp22 [8/256/0/180] [ 7096.793878] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7096.807080] LNet: Added LNI 192.168.202.116@tcp23 [8/256/0/180] [ 7098.645647] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7098.655488] LNet: Added LNI 192.168.202.116@tcp24 [8/256/0/180] [ 7100.314897] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7100.327346] LNet: Added LNI 192.168.202.116@tcp25 [8/256/0/180] [ 7101.974561] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7101.997580] LNet: Added LNI 192.168.202.116@tcp26 [8/256/0/180] [ 7104.182799] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7104.194916] LNet: Added LNI 192.168.202.116@tcp27 [8/256/0/180] [ 7106.067290] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7106.079361] LNet: Added LNI 192.168.202.116@tcp28 [8/256/0/180] [ 7108.096143] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7108.105485] LNet: Added LNI 192.168.202.116@tcp29 [8/256/0/180] [ 7110.263140] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7110.273020] LNet: Added LNI 192.168.202.116@tcp30 [8/256/0/180] [ 7112.225601] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7112.234685] LNet: Added LNI 192.168.202.116@tcp31 [8/256/0/180] [ 7113.925854] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7113.934444] LNet: Added LNI 192.168.202.116@tcp32 [8/256/0/180] [ 7115.864050] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7115.874623] LNet: Added LNI 192.168.202.116@tcp33 [8/256/0/180] [ 7117.716314] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7117.725380] LNet: Added LNI 192.168.202.116@tcp34 [8/256/0/180] [ 7119.493298] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7119.502458] LNet: Added LNI 192.168.202.116@tcp35 [8/256/0/180] [ 7121.291778] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7121.300464] LNet: Added LNI 192.168.202.116@tcp36 [8/256/0/180] [ 7123.127462] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7123.136532] LNet: Added LNI 192.168.202.116@tcp37 [8/256/0/180] [ 7125.002259] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7125.012209] LNet: Added LNI 192.168.202.116@tcp38 [8/256/0/180] [ 7126.955405] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7126.963862] LNet: Added LNI 192.168.202.116@tcp39 [8/256/0/180] [ 7128.832140] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7128.840832] LNet: Added LNI 192.168.202.116@tcp40 [8/256/0/180] [ 7131.074726] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7131.087169] LNet: Added LNI 192.168.202.116@tcp41 [8/256/0/180] [ 7133.491579] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7133.503111] LNet: Added LNI 192.168.202.116@tcp42 [8/256/0/180] [ 7135.734380] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7135.746304] LNet: Added LNI 192.168.202.116@tcp43 [8/256/0/180] [ 7138.157342] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7138.172628] LNet: Added LNI 192.168.202.116@tcp44 [8/256/0/180] [ 7141.063991] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7141.072891] LNet: Added LNI 192.168.202.116@tcp45 [8/256/0/180] [ 7143.846145] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7143.861785] LNet: Added LNI 192.168.202.116@tcp46 [8/256/0/180] [ 7146.420254] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7146.433975] LNet: Added LNI 192.168.202.116@tcp47 [8/256/0/180] [ 7149.196138] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7149.211691] LNet: Added LNI 192.168.202.116@tcp48 [8/256/0/180] [ 7151.521362] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7151.534484] LNet: Added LNI 192.168.202.116@tcp49 [8/256/0/180] [ 7154.330826] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7154.342814] LNet: Added LNI 192.168.202.116@tcp50 [8/256/0/180] [ 7165.920489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7165.926106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.934772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7166.434453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7166.435411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.438239] Lustre: Skipped 1 previous similar message [ 7166.445256] Lustre: Skipped 1 previous similar message [ 7169.114900] Lustre: server umount lustre-MDT0000 complete [ 7170.334884] LustreError: 443253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773948663 with bad export cookie 5021013098361287182 [ 7170.340237] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7170.340721] LustreError: 443253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7170.465808] Lustre: server umount lustre-MDT0001 complete [ 7178.148526] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7181.489526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7187.935250] Lustre: 442531:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773948665/real 1773948665] req@0000000088ee71e6 x1860119854924672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773948681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7187.955105] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7192.095088] Lustre: 442531:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773948669/real 1773948669] req@00000000396dc16a x1860119854924800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773948685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7195.167300] LustreError: 442530:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000051a995ec x1860119854925824/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7195.275750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.319118] Lustre: 442534:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773948673/real 1773948673] req@00000000b09455c1 x1860119854925184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773948689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7196.325520] Lustre: 442534:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7196.476669] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7199.824738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7199.957907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7200.927090] Lustre: 442531:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773948678/real 1773948678] req@0000000097a5a3cf x1860119854925312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773948694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7201.126223] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7203.563245] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7205.114765] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7205.351652] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7207.402329] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7215.585787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7215.593050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.605419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7215.609543] Lustre: Skipped 1 previous similar message [ 7217.637419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7217.643313] Lustre: Skipped 2 previous similar messages [ 7219.722592] Lustre: server umount lustre-OST0000 complete [ 7222.752613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7222.753255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7222.755458] Lustre: Skipped 2 previous similar messages [ 7222.758135] Lustre: Skipped 1 previous similar message [ 7227.116582] Lustre: server umount lustre-MDT0000 complete [ 7227.873586] LustreError: 468395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7227.880106] LustreError: 468395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7228.754940] LustreError: 443254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773948722 with bad export cookie 5021013098361289205 [ 7228.761927] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7228.767679] LustreError: 443254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7228.904701] Lustre: server umount lustre-MDT0001 complete [ 7233.438453] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7234.922771] Key type lgssc unregistered [ 7235.143459] LNet: 470925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7235.149708] LNetError: 470925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7235.154427] LNet: Removed LNI 192.168.202.116@tcp [ 7235.157061] LNet: Removed LNI 192.168.202.116@tcp10 [ 7235.159564] LNet: Removed LNI 192.168.202.116@tcp11 [ 7235.162190] LNet: Removed LNI 192.168.202.116@tcp12 [ 7235.164738] LNet: Removed LNI 192.168.202.116@tcp13 [ 7235.167227] LNet: Removed LNI 192.168.202.116@tcp14 [ 7235.169624] LNet: Removed LNI 192.168.202.116@tcp15 [ 7235.172054] LNet: Removed LNI 192.168.202.116@tcp16 [ 7235.174453] LNet: Removed LNI 192.168.202.116@tcp17 [ 7235.176677] LNet: Removed LNI 192.168.202.116@tcp18 [ 7235.180252] LNet: Removed LNI 192.168.202.116@tcp19 [ 7235.182479] LNet: Removed LNI 192.168.202.116@tcp20 [ 7235.184732] LNet: Removed LNI 192.168.202.116@tcp21 [ 7235.186968] LNet: Removed LNI 192.168.202.116@tcp22 [ 7235.189247] LNet: Removed LNI 192.168.202.116@tcp23 [ 7235.191318] LNet: Removed LNI 192.168.202.116@tcp24 [ 7235.193362] LNet: Removed LNI 192.168.202.116@tcp25 [ 7235.195410] LNet: Removed LNI 192.168.202.116@tcp26 [ 7235.197430] LNet: Removed LNI 192.168.202.116@tcp27 [ 7235.199459] LNet: Removed LNI 192.168.202.116@tcp28 [ 7235.201319] LNet: Removed LNI 192.168.202.116@tcp29 [ 7235.203127] LNet: Removed LNI 192.168.202.116@tcp30 [ 7235.205038] LNet: Removed LNI 192.168.202.116@tcp31 [ 7235.206904] LNet: Removed LNI 192.168.202.116@tcp32 [ 7235.208613] LNet: Removed LNI 192.168.202.116@tcp33 [ 7235.210318] LNet: Removed LNI 192.168.202.116@tcp34 [ 7235.211976] LNet: Removed LNI 192.168.202.116@tcp35 [ 7235.213614] LNet: Removed LNI 192.168.202.116@tcp36 [ 7235.215346] LNet: Removed LNI 192.168.202.116@tcp37 [ 7235.217138] LNet: Removed LNI 192.168.202.116@tcp38 [ 7235.218655] LNet: Removed LNI 192.168.202.116@tcp39 [ 7235.220207] LNet: Removed LNI 192.168.202.116@tcp40 [ 7235.221834] LNet: Removed LNI 192.168.202.116@tcp41 [ 7235.223375] LNet: Removed LNI 192.168.202.116@tcp42 [ 7235.224949] LNet: Removed LNI 192.168.202.116@tcp43 [ 7235.226402] LNet: Removed LNI 192.168.202.116@tcp44 [ 7235.227945] LNet: Removed LNI 192.168.202.116@tcp45 [ 7235.229351] LNet: Removed LNI 192.168.202.116@tcp46 [ 7235.230885] LNet: Removed LNI 192.168.202.116@tcp47 [ 7235.232350] LNet: Removed LNI 192.168.202.116@tcp48 [ 7235.233839] LNet: Removed LNI 192.168.202.116@tcp49 [ 7235.239724] LNet: Removed LNI 192.168.202.116@tcp50 [ 7235.682695] Key type .llcrypt unregistered [ 7235.685070] Key type ._llcrypt unregistered [ 7244.857688] Key type ._llcrypt registered [ 7244.858906] Key type .llcrypt registered [ 7244.926351] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7245.344355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7245.390082] alg: No test for adler32 (adler32-zlib) [ 7246.272703] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 7246.381401] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7247.991123] Key type lgssc registered [ 7248.623777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7252.522646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7252.533475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.774204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.662672] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7259.954132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7260.123945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7261.694859] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7263.952565] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7265.623575] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7268.822927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7268.993566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7271.223121] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7273.760611] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7279.271088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7288.925236] loop0: detected capacity change from 0 to 204800000 [ 7289.035863] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7289.042159] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7289.048486] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7291.945278] loop0: detected capacity change from 0 to 204800000 [ 7292.015117] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7292.018617] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7292.021903] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7294.403600] loop0: detected capacity change from 0 to 204800000 [ 7296.879784] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7296.941170] Lustre: fs15246-OST0000: new disk, initializing [ 7296.944413] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7296.974776] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7299.456131] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7304.618766] loop1: detected capacity change from 0 to 204800000 [ 7307.146866] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7307.203255] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7307.234767] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7307.287618] Lustre: fs15246-MDT0000: new disk, initializing [ 7307.367890] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.382706] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7307.461186] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7308.511787] Lustre: 476528:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000fe82cb1a x1860120072980608/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773948818 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7308.978520] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7317.602261] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7317.605462] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7317.614715] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7321.061384] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7321.068190] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.771380] Lustre: server umount fs15246-MDT0000 complete [ 7326.181702] LustreError: 472142:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.189557] LustreError: 472142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7329.060781] Lustre: server umount fs15246-OST0000 complete [ 7335.904604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7335.907877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.914113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7341.537440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7341.540584] Lustre: Skipped 3 previous similar messages [ 7346.657328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7346.660157] Lustre: Skipped 1 previous similar message [ 7347.167137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7347.234190] Lustre: server umount lustre-OST0000 complete [ 7351.777663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.779954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.784407] Lustre: Skipped 2 previous similar messages [ 7354.723918] Lustre: server umount lustre-MDT0000 complete [ 7356.380885] LustreError: 476521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773948849 with bad export cookie 10874498947756338987 [ 7356.386214] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7356.386968] LustreError: 476521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7356.512571] Lustre: server umount lustre-MDT0001 complete [ 7360.132450] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7361.517208] Key type lgssc unregistered [ 7361.692583] LNet: 479591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7361.697300] LNetError: 479591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7361.707585] LNet: Removed LNI 192.168.202.116@tcp [ 7362.164233] Key type .llcrypt unregistered [ 7362.167669] Key type ._llcrypt unregistered [ 7369.914349] Key type ._llcrypt registered [ 7369.915260] Key type .llcrypt registered [ 7369.951894] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7370.225957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7370.232340] alg: No test for adler32 (adler32-zlib) [ 7371.070654] Lustre: Lustre: Build Version: 2.17.50_193_gf5bca74 [ 7371.151638] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7372.727160] Key type lgssc registered [ 7373.286288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7389.090663] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7393.250419] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7405.653315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7405.662928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.797854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7406.815471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7406.860886] Lustre: lustre-MDT0000: new disk, initializing [ 7406.885408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7406.891378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7408.327582] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7413.549092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.600373] Lustre: 483978:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7413.636894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7413.641376] Lustre: Skipped 1 previous similar message [ 7413.689046] Lustre: lustre-MDT0001: new disk, initializing [ 7413.726978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7413.742858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7413.746539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7415.215250] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7418.854468] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7420.351982] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7423.196770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7423.310211] Lustre: lustre-OST0000: new disk, initializing [ 7423.311925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7423.339105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7425.232993] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7428.089130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7428.093644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7428.131238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7428.358901] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7429.993082] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7430.065239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7431.812755] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7431.898013] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7433.187591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7433.188530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.189528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.190616] LustreError: Skipped 1 previous similar message [ 7433.199222] Lustre: Skipped 1 previous similar message [ 7434.846384] Lustre: server umount lustre-OST0000 complete [ 7438.306329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.307667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.313503] Lustre: Skipped 1 previous similar message [ 7438.317264] Lustre: Skipped 1 previous similar message [ 7442.797310] Lustre: server umount lustre-MDT0000 complete [ 7444.601915] LustreError: 483971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773948938 with bad export cookie 18308323817646618606 [ 7444.609088] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7444.611994] LustreError: 483971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7444.752803] Lustre: server umount lustre-MDT0001 complete [ 7455.463611] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7459.084886] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7472.193326] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7354 sec ========== 15:36:04 (1773948964) [ 7472.971624] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:36:04 (1773948964) === [ 7474.615984] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:36:06 (1773948966) === [ 7489.849357] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7491.082675] Key type lgssc unregistered [ 7491.211476] LNet: 492152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7491.214916] LNetError: 492152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7491.223512] LNet: Removed LNI 192.168.202.116@tcp [ 7491.607623] Key type .llcrypt unregistered [ 7491.609460] Key type ._llcrypt unregistered