[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 544602833 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002339] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.006974] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009151] LSM: Security Framework initializing [ 0.010062] Yama: becoming mindful. [ 0.011043] SELinux: Initializing. [ 0.013024] *** VALIDATE selinux *** [ 0.021827] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026620] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029123] *** VALIDATE tmpfs *** [ 0.031197] *** VALIDATE proc *** [ 0.032257] *** VALIDATE cgroup *** [ 0.033015] *** VALIDATE cgroup2 *** [ 0.034299] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036119] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038040] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042304] debug: unmapping init [mem 0xffffffffa3859000-0xffffffffa3860fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050021] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053314] rcu: Hierarchical SRCU implementation. [ 0.055510] smp: Bringing up secondary CPUs ... [ 0.056597] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.064025] smp: Brought up 1 node, 4 CPUs [ 0.066014] smpboot: Max logical packages: 1 [ 0.067014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.254090] node 0 deferred pages initialised in 185ms [ 0.258106] devtmpfs: initialized [ 0.259376] x86/mm: Memory block size: 128MB [ 0.263108] gcov: version magic: 0x41383552 [ 0.267635] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.268115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269513] pinctrl core: initialized pinctrl subsystem [ 0.271274] [ 0.271929] ************************************************************* [ 0.275027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277016] ** ** [ 0.280017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.284024] ** ** [ 0.287015] ** This means that this kernel is built to expose internal ** [ 0.289019] ** IOMMU data structures, which may compromise security on ** [ 0.292022] ** your system. ** [ 0.295019] ** ** [ 0.298019] ** If you see this message and you are not debugging the ** [ 0.300029] ** kernel, report this immediately to your vendor! ** [ 0.303023] ** ** [ 0.306032] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.309034] ************************************************************* [ 0.313170] NET: Registered protocol family 16 [ 0.315553] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.318105] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.321092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.325040] cpuidle: using governor menu [ 0.327000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.330692] PCI: Using configuration type 1 for base access [ 0.333164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.345113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.348026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.355047] cryptd: max_cpu_qlen set to 1000 [ 0.359322] ACPI: Added _OSI(Module Device) [ 0.361041] ACPI: Added _OSI(Processor Device) [ 0.363022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.365020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.369736] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.377640] ACPI: Interpreter enabled [ 0.379073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.381017] ACPI: Using IOAPIC for interrupt routing [ 0.383184] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.387464] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.398183] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.400051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.403029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.406111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.411579] acpiphp: Slot [2] registered [ 0.414186] acpiphp: Slot [5] registered [ 0.415171] acpiphp: Slot [6] registered [ 0.417127] acpiphp: Slot [7] registered [ 0.418137] acpiphp: Slot [8] registered [ 0.420151] acpiphp: Slot [9] registered [ 0.421141] acpiphp: Slot [10] registered [ 0.422119] acpiphp: Slot [3] registered [ 0.424115] acpiphp: Slot [4] registered [ 0.426117] acpiphp: Slot [11] registered [ 0.427127] acpiphp: Slot [12] registered [ 0.429104] acpiphp: Slot [13] registered [ 0.430109] acpiphp: Slot [14] registered [ 0.431150] acpiphp: Slot [15] registered [ 0.432086] acpiphp: Slot [16] registered [ 0.434113] acpiphp: Slot [17] registered [ 0.436123] acpiphp: Slot [18] registered [ 0.437104] acpiphp: Slot [19] registered [ 0.439122] acpiphp: Slot [20] registered [ 0.440146] acpiphp: Slot [21] registered [ 0.442145] acpiphp: Slot [22] registered [ 0.443122] acpiphp: Slot [23] registered [ 0.445153] acpiphp: Slot [24] registered [ 0.447172] acpiphp: Slot [25] registered [ 0.448114] acpiphp: Slot [26] registered [ 0.450171] acpiphp: Slot [27] registered [ 0.452139] acpiphp: Slot [28] registered [ 0.453175] acpiphp: Slot [29] registered [ 0.455120] acpiphp: Slot [30] registered [ 0.457182] acpiphp: Slot [31] registered [ 0.458071] PCI host bridge to bus 0000:00 [ 0.460026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.462029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.465030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.467025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.470036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.473031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.475214] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.478251] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.482498] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.492767] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.499064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.502027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.505036] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.510035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.513362] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.515763] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.518048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.521914] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.528026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.542033] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.548019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.555388] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.569024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.595022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.623021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.641796] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.650023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.664025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.697021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.711474] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.719022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.728022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.753022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.765907] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.775025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.784026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.811021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.823955] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.833020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.841025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.866020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.882000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.891022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.898021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.926024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.942214] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.945402] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.948508] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.951520] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.954617] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.959175] iommu: Default domain type: Passthrough [ 0.962688] SCSI subsystem initialized [ 0.964234] ACPI: bus type USB registered [ 0.967248] usbcore: registered new interface driver usbfs [ 0.970128] usbcore: registered new interface driver hub [ 0.972092] usbcore: registered new device driver usb [ 0.974216] pps_core: LinuxPPS API ver. 1 registered [ 0.976028] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.980210] PTP clock support registered [ 0.982185] EDAC MC: Ver: 3.0.0 [ 0.985064] PCI: Using ACPI for IRQ routing [ 0.987073] NetLabel: Initializing [ 0.989014] NetLabel: domain hash size = 128 [ 0.990015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.993109] NetLabel: unlabeled traffic allowed by default [ 0.995194] vgaarb: loaded [ 0.997334] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.999020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.005839] clocksource: Switched to clocksource kvm-clock [ 1.135855] VFS: Disk quotas dquot_6.6.0 [ 1.137745] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.140420] *** VALIDATE ramfs *** [ 1.141874] *** VALIDATE hugetlbfs *** [ 1.143510] pnp: PnP ACPI init [ 1.146234] pnp: PnP ACPI: found 6 devices [ 1.170115] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.173340] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.175542] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.178059] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.181356] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.184293] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.187722] NET: Registered protocol family 2 [ 1.190598] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.196421] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.200809] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.207932] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.211279] TCP: Hash tables configured (established 65536 bind 65536) [ 1.213453] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.216361] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.219799] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.223411] NET: Registered protocol family 1 [ 1.226373] RPC: Registered named UNIX socket transport module. [ 1.229873] RPC: Registered udp transport module. [ 1.232108] RPC: Registered tcp transport module. [ 1.234332] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.237911] NET: Registered protocol family 44 [ 1.239830] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.242249] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.244708] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.247296] PCI: CLS 0 bytes, default 64 [ 1.249181] Unpacking initramfs... [ 2.794311] debug: unmapping init [mem 0xffff9d8a3cc54000-0xffff9d8a3ffbffff] [ 2.800772] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.803289] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.807483] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.352316] Initialise system trusted keyrings [ 3.354628] Key type blacklist registered [ 3.356729] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.366853] zbud: loaded [ 3.370127] *** VALIDATE nfs *** [ 3.371576] *** VALIDATE nfs4 *** [ 3.373364] pstore: using deflate compression [ 3.377094] Platform Keyring initialized [ 3.504742] NET: Registered protocol family 38 [ 3.506861] Key type asymmetric registered [ 3.508562] Asymmetric key parser 'x509' registered [ 3.510759] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.514280] io scheduler mq-deadline registered [ 3.516011] io scheduler kyber registered [ 3.517919] io scheduler bfq registered [ 3.520661] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.523929] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.527723] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.533467] ACPI: Power Button [PWRF] [ 3.539162] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.546660] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.563358] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.576632] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.599049] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.627624] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.658518] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.663756] Non-volatile memory driver v1.3 [ 3.665539] Linux agpgart interface v0.103 [ 3.696574] virtio_blk virtio1: [vda] 145960 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.701148] vda: detected capacity change from 0 to 74731520 [ 3.718546] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.724454] vdb: detected capacity change from 0 to 1073741824 [ 3.752320] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.755230] vdc: detected capacity change from 0 to 2621440000 [ 3.770792] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.774127] vdd: detected capacity change from 0 to 2621440000 [ 3.795677] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.798883] vde: detected capacity change from 0 to 4294967296 [ 3.817758] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.821121] vdf: detected capacity change from 0 to 4294967296 [ 3.828253] libphy: Fixed MDIO Bus: probed [ 3.841325] usbcore: registered new interface driver usbserial_generic [ 3.843964] usbserial: USB Serial support registered for generic [ 3.846827] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.850907] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.852656] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.855877] mousedev: PS/2 mouse device common for all mice [ 3.858890] rtc_cmos 00:05: RTC can wake from S4 [ 3.861487] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.861936] rtc_cmos 00:05: registered as rtc0 [ 3.866684] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.869749] intel_pstate: CPU model not supported [ 3.870489] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.876249] hid: raw HID events driver (C) Jiri Kosina [ 3.878309] usbcore: registered new interface driver usbhid [ 3.878328] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.880145] usbhid: USB HID core driver [ 3.885124] drop_monitor: Initializing network drop monitor service [ 3.887139] Initializing XFRM netlink socket [ 3.891885] NET: Registered protocol family 10 [ 3.895272] Segment Routing with IPv6 [ 3.896912] NET: Registered protocol family 17 [ 3.899118] mpls_gso: MPLS GSO support [ 3.906486] RAS: Correctable Errors collector initialized. [ 3.908725] AVX version of gcm_enc/dec engaged. [ 3.911726] AES CTR mode by8 optimization enabled [ 3.994647] sched_clock: Marking stable (3994626338, 0)->(4945663557, -951037219) [ 3.999294] registered taskstats version 1 [ 4.002231] Loading compiled-in X.509 certificates [ 4.004600] zswap: loaded using pool lzo/zbud [ 4.033254] Key type big_key registered [ 4.052648] Key type encrypted registered [ 4.054849] ima: No TPM chip found, activating TPM-bypass! [ 4.057677] ima: Allocated hash algorithm: sha1 [ 4.059776] ima: No architecture policies found [ 4.062031] evm: Initialising EVM extended attributes: [ 4.064435] evm: security.selinux [ 4.065914] evm: security.ima [ 4.066987] evm: security.capability [ 4.068597] evm: HMAC attrs: 0x1 [ 4.073162] rtc_cmos 00:05: setting system clock to 2026-09-08 21:29:18 UTC (1788902958) [ 4.079603] debug: unmapping init [mem 0xffffffffa4803000-0xffffffffa49fffff] [ 4.083023] debug: unmapping init [mem 0xffffffffa3582000-0xffffffffa3858fff] [ 4.092066] Write protecting the kernel read-only data: 28672k [ 4.096140] debug: unmapping init [mem 0xffffffffa1c03000-0xffffffffa1dfffff] [ 4.099350] debug: unmapping init [mem 0xffffffffa2514000-0xffffffffa25fffff] [ 4.138180] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.147396] systemd[1]: Detected virtualization kvm. [ 4.149636] systemd[1]: Detected architecture x86-64. [ 4.151473] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.175165] systemd[1]: No hostname configured. [ 4.177146] systemd[1]: Set hostname to . [ 4.179373] random: systemd: uninitialized urandom read (16 bytes read) [ 4.182194] systemd[1]: Initializing machine ID from random generator. [ 4.209078] random: ln: uninitialized urandom read (6 bytes read) [ 4.338990] random: systemd: uninitialized urandom read (16 bytes read) [ 4.341314] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.345162] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.351405] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.942527] device-mapper: uevent: version 1.0.3 [ 4.945862] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.647073] virtio_net virtio0 ens2: renamed from eth0 [ 5.663436] random: fast init done [ 5.690400] scsi host0: ata_piix [ 5.713199] scsi host1: ata_piix [ 5.714802] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.717679] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.435576] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.581649] random: crng init done [ 10.582861] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.956857] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.073310] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.361020] SELinux: Disabled at runtime. [ 12.421713] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.431174] systemd[1]: Detected virtualization kvm. [ 12.433052] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.936442] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.939866] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.944537] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.950301] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.954061] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.961296] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.966670] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 13.108566] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.450408] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.750566] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.773167] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.866310] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.884469] EDAC sbridge: Ver: 1.1.2 [ 15.342996] Key type dns_resolver registered [ 15.651514] NFS: Registering the id_resolver key type [ 15.653856] Key type id_resolver registered [ 15.655532] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg415-server login: [ 42.973019] hrtimer: interrupt took 7007882 ns [ 50.139175] libcfs: loading out-of-tree module taints kernel. [ 50.180254] Key type ._llcrypt registered [ 50.182183] Key type .llcrypt registered [ 50.264582] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 67.523961] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 68.965465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.994636] alg: No test for adler32 (adler32-zlib) [ 70.695846] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 71.689472] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 73.473136] Key type lgssc registered [ 75.697991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.945882] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 137.895454] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 149.897771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.954885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.194573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 151.263930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 151.342971] Lustre: lustre-MDT0000: new disk, initializing [ 151.421643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.443164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.415835] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.708518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.840982] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.869855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.873343] Lustre: Skipped 1 previous similar message [ 169.952837] Lustre: lustre-MDT0001: new disk, initializing [ 170.053860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.089026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.109280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.369727] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.854055] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.826871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.978798] Lustre: lustre-OST0000: new disk, initializing [ 190.982552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.986190] Lustre: 8442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.046186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.842819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.847635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.919434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.705677] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.344703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.558811] Lustre: lustre-OST0001: new disk, initializing [ 211.570740] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.577367] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.692757] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.817272] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.756051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 221.767838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 221.866251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.093419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.844503] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.105748] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.359286] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing yml_node [ 260.381691] Lustre: DEBUG MARKER: Client: 2.17.55.26 [ 263.060411] Lustre: DEBUG MARKER: MDS: 2.17.55.26 [ 266.049277] Lustre: DEBUG MARKER: OSS: 2.17.55.26 [ 268.102452] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:33:41 EDT 2026 [ 288.662876] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 290.718603] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 308.193237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 308.203450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.228116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.221251] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.225077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.253304] Lustre: Skipped 2 previous similar messages [ 309.270807] Lustre: Skipped 3 previous similar messages [ 313.921059] Lustre: server umount lustre-MDT0000 complete [ 317.537848] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903271 with bad export cookie 527948120722222230 [ 317.537997] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 317.547451] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 319.456231] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.461715] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.463157] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.470631] Lustre: Skipped 1 previous similar message [ 319.497867] LustreError: 6510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 322.466638] LustreError: 7459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.478082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.494207] Lustre: Skipped 2 previous similar messages [ 323.887952] Lustre: server umount lustre-MDT0001 complete [ 328.232465] Lustre: server umount lustre-OST0000 complete [ 332.322689] Lustre: server umount lustre-OST0001 complete [ 340.355920] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 350.352648] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 397.530449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.990597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 398.076833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 398.273325] Lustre: lustre-MDT0000: new disk, initializing [ 398.439184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.464770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 404.150295] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 417.773502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.029955] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 418.196846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 418.217834] Lustre: Skipped 1 previous similar message [ 418.381278] Lustre: lustre-MDT0001: new disk, initializing [ 418.537338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 418.618267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 418.632699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.122209] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 437.267913] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 445.255553] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 454.309848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 454.788721] Lustre: lustre-OST0000: new disk, initializing [ 454.794940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 454.807155] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 454.958026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.940235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 455.954269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 456.108805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 461.375845] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 470.156559] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 474.794667] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 474.996068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.735267] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 478.935736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.761966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.766607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.769529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.787913] Lustre: Skipped 1 previous similar message [ 486.766150] Lustre: server umount lustre-OST0000 complete [ 490.465340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 490.473888] LustreError: Skipped 1 previous similar message [ 490.480731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.497514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 490.505193] Lustre: Skipped 1 previous similar message [ 495.919602] Lustre: server umount lustre-MDT0000 complete [ 497.126027] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.137767] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 498.977373] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903453 with bad export cookie 527948120722223763 [ 498.989858] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.993900] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 499.240619] Lustre: server umount lustre-MDT0001 complete [ 502.769946] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:37:36 (1788903456) [ 504.964951] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 506.256744] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:37:39 (1788903459) [ 511.542953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.850904] LustreError: 19449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.914870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.663313] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 517.091443] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.485048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.300540] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 526.755451] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 529.597614] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 533.426113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.930570] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 540.812496] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 551.394904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.399125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.399156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 551.403031] Lustre: Skipped 2 previous similar messages [ 551.410660] Lustre: Skipped 3 previous similar messages [ 554.745700] Lustre: server umount lustre-OST0000 complete [ 561.645956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 571.359394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.443071] Lustre: server umount lustre-MDT0000 complete [ 573.483359] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903527 with bad export cookie 527948120722224645 [ 573.483988] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.491674] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 573.684237] Lustre: server umount lustre-MDT0001 complete [ 578.818856] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 580.223415] Key type lgssc unregistered [ 580.394734] LNet: 22980:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 580.399317] LNetError: 22980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 580.412349] LNet: Removed LNI 192.168.204.115@tcp [ 580.871173] Key type .llcrypt unregistered [ 580.873370] Key type ._llcrypt unregistered [ 592.049701] Key type ._llcrypt registered [ 592.051134] Key type .llcrypt registered [ 592.108079] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 592.662934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 592.676868] alg: No test for adler32 (adler32-zlib) [ 593.587654] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 593.719262] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 595.319164] Key type lgssc registered [ 595.853439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.569891] loop: module loaded [ 599.599217] loop0: detected capacity change from 0 to 204800000 [ 599.690923] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 599.695347] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 599.835489] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.516656] loop0: detected capacity change from 0 to 204800000 [ 602.642791] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.649294] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.655679] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.175825] loop0: detected capacity change from 0 to 204800000 [ 607.901539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 607.912622] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.031821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 609.057876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 609.102315] Lustre: lustre-MDT0000: new disk, initializing [ 609.155153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.166884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 611.204663] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 616.343696] loop1: detected capacity change from 0 to 204800000 [ 619.280840] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 619.423314] Lustre: lustre-OST07c6: new disk, initializing [ 619.426243] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 619.430392] Lustre: Skipped 1 previous similar message [ 619.434978] Lustre: 26139:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 619.474276] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 622.370157] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 627.740352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 627.745555] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 627.772140] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 630.383237] Lustre: 26655:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 630.389057] Lustre: 26655:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 630.392032] Lustre: 26655:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 630.395535] Lustre: 26655:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 630.400023] Lustre: 26655:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 630.405270] Lustre: 26655:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 645.140046] Lustre: 26655:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 645.145886] Lustre: 26655:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 645.149705] Lustre: 26655:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 645.153673] Lustre: 26655:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 645.157529] Lustre: 26655:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 645.161197] Lustre: 26655:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 649.094772] Lustre: 26655:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 649.101809] Lustre: 26655:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 649.106485] Lustre: 26655:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 649.110350] Lustre: 26655:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 649.114026] Lustre: 26655:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 649.122286] Lustre: 26655:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 649.127508] Lustre: 26655:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 649.132541] Lustre: 26655:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 649.139860] Lustre: 26655:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 649.146267] Lustre: 26655:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 649.150403] Lustre: 26655:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 649.153984] Lustre: 26655:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.252376] Lustre: 25129:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 652.257538] Lustre: 25129:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 652.261787] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 652.265585] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.271343] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 652.276391] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.281510] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 652.286659] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.298027] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 652.304019] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.308218] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 652.311754] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.187438] Lustre: 25127:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 659.191590] Lustre: 25127:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 659.195405] Lustre: 25127:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 659.198756] Lustre: 25127:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 659.203455] Lustre: 25127:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 659.207048] Lustre: 25127:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 659.210477] Lustre: 25127:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 659.214573] Lustre: 25127:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 659.218225] Lustre: 25127:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 659.221686] Lustre: 25127:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 659.224701] Lustre: 25127:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 659.227780] Lustre: 25127:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 669.117307] Lustre: 25128:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 669.120988] Lustre: 25128:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 669.123744] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 669.127435] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 669.131163] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 669.135075] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 669.139603] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 669.143886] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 669.147576] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 669.151227] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 669.154800] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 669.158446] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 713.583854] Lustre: server umount lustre-OST07c6 complete [ 728.139423] Lustre: server umount lustre-MDT0000 complete [ 735.264855] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 736.496218] Key type lgssc unregistered [ 736.639830] LNet: 28719:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.646727] LNetError: 28719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.656416] LNet: Removed LNI 192.168.204.115@tcp [ 737.042540] Key type .llcrypt unregistered [ 737.044113] Key type ._llcrypt unregistered [ 745.666608] Key type ._llcrypt registered [ 745.667862] Key type .llcrypt registered [ 745.705145] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 746.149161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 746.157428] alg: No test for adler32 (adler32-zlib) [ 747.021282] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 747.122426] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 748.711337] Key type lgssc registered [ 749.163484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 752.466856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 752.478450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.670019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.349219] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 758.832394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.986585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 760.718483] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 763.606960] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 765.820118] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 768.454561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 768.616415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.916226] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 773.928254] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 778.858987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 787.940074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 787.944280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 787.952105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 789.474205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 789.475785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.519214] Lustre: server umount lustre-OST0000 complete [ 794.592681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 794.593502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.598233] Lustre: Skipped 1 previous similar message [ 794.600948] Lustre: Skipped 1 previous similar message [ 799.712229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.714619] Lustre: Skipped 2 previous similar messages [ 804.832268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 804.835740] Lustre: Skipped 1 previous similar message [ 807.391159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 807.461435] Lustre: server umount lustre-MDT0000 complete [ 808.866208] LustreError: 32661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903763 with bad export cookie 2749042299082651251 [ 808.871039] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 808.871046] LustreError: 32661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 808.994742] Lustre: server umount lustre-MDT0001 complete [ 813.126399] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 814.307089] Key type lgssc unregistered [ 814.454466] LNet: 33702:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.457942] LNetError: 33702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.468409] LNet: Removed LNI 192.168.204.115@tcp [ 814.825099] Key type .llcrypt unregistered [ 814.826515] Key type ._llcrypt unregistered [ 822.952597] Key type ._llcrypt registered [ 822.954394] Key type .llcrypt registered [ 822.999423] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 823.313774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.324396] alg: No test for adler32 (adler32-zlib) [ 824.203622] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 824.296315] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 825.887173] Key type lgssc registered [ 826.279444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 829.183259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 829.192492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.368873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.930258] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 835.638639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.813275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 837.432267] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 840.318960] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 842.576290] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 845.342503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 845.496982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.665860] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 850.469273] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 854.694759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 860.128781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 860.131945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.133077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.137705] Lustre: Skipped 1 previous similar message [ 865.249570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 865.251972] Lustre: Skipped 2 previous similar messages [ 865.953831] Lustre: server umount lustre-OST0000 complete [ 870.369752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.370536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.374608] Lustre: Skipped 1 previous similar message [ 873.486466] Lustre: server umount lustre-MDT0000 complete [ 874.702377] LustreError: 34928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903829 with bad export cookie 1598575534663410662 [ 874.706151] LustreError: 34928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 874.707486] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.822178] Lustre: server umount lustre-MDT0001 complete [ 878.284897] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 879.283978] Key type lgssc unregistered [ 879.415368] LNet: 38478:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.417527] LNetError: 38478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.429372] LNet: Removed LNI 192.168.204.115@tcp [ 879.729643] Key type .llcrypt unregistered [ 879.730651] Key type ._llcrypt unregistered [ 888.399245] Key type ._llcrypt registered [ 888.400894] Key type .llcrypt registered [ 888.440222] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 888.790764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 888.840483] alg: No test for adler32 (adler32-zlib) [ 889.682287] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 889.767570] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 891.343133] Key type lgssc registered [ 891.690773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.123321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.129272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.273091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.512552] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 899.211816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.330098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 900.579613] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 902.881787] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 904.651334] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.938099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.051387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 908.977086] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 911.316311] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 921.382853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 925.151889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 925.154375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.158634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.689928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.689949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.696810] Lustre: Skipped 2 previous similar messages [ 929.941460] Lustre: server umount lustre-MDT0000 complete [ 931.123137] LustreError: 40603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903885 with bad export cookie 7988625045120756686 [ 931.126558] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.127565] LustreError: 40603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 931.807822] LustreError: 39866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.808045] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.812427] LustreError: 39866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 931.815548] Lustre: Skipped 1 previous similar message [ 931.816092] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 931.820917] Lustre: Skipped 1 previous similar message [ 936.927753] LustreError: 39866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 936.928072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 936.932585] LustreError: 39866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 936.937872] Lustre: Skipped 1 previous similar message [ 937.320773] Lustre: server umount lustre-MDT0001 complete [ 955.087345] Lustre: server umount lustre-OST0000 complete [ 960.340292] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 961.387101] Key type lgssc unregistered [ 961.510323] LNet: 43748:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.512584] LNetError: 43748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.523304] LNet: Removed LNI 192.168.204.115@tcp [ 961.824317] Key type .llcrypt unregistered [ 961.825491] Key type ._llcrypt unregistered [ 970.342419] Key type ._llcrypt registered [ 970.343755] Key type .llcrypt registered [ 970.384379] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 970.792066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 970.808228] alg: No test for adler32 (adler32-zlib) [ 971.661097] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 971.753818] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 973.335147] Key type lgssc registered [ 973.704736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 976.252242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 976.258406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.397846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.661712] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 981.378658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.495669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.754143] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 985.052680] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 986.756252] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 989.045466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 989.158642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.025325] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 993.455378] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1002.469496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1006.559858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1006.563045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.568718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.584826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.584874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.585183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.146059] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1009.148410] Lustre: Skipped 1 previous similar message [ 1012.301983] Lustre: server umount lustre-OST0000 complete [ 1024.905616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.906539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1024.909576] Lustre: Skipped 1 previous similar message [ 1028.960062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.962997] Lustre: Skipped 2 previous similar messages [ 1038.304473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.307638] Lustre: Skipped 3 previous similar messages [ 1039.327097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1039.381644] Lustre: server umount lustre-MDT0000 complete [ 1040.564172] LustreError: 45119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903994 with bad export cookie 9238789699875932791 [ 1040.567668] LustreError: 45119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1040.568673] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1046.695792] Lustre: server umount lustre-MDT0001 complete [ 1050.000014] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1050.914523] Key type lgssc unregistered [ 1051.035281] LNet: 48863:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.037834] LNetError: 48863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.043242] LNet: Removed LNI 192.168.204.115@tcp [ 1051.340225] Key type .llcrypt unregistered [ 1051.342039] Key type ._llcrypt unregistered [ 1059.592861] Key type ._llcrypt registered [ 1059.593790] Key type .llcrypt registered [ 1059.624468] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1059.931453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.963657] alg: No test for adler32 (adler32-zlib) [ 1060.809990] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1060.889243] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1062.463130] Key type lgssc registered [ 1062.781769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.154481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.159791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.286314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.509510] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1070.099587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.207119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.465079] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1073.630157] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1075.306097] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1077.586133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.707819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.631779] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1081.893417] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1092.071519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1094.604348] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1095.751284] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1106.401720] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0817h47m40s [ 1106.857225] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1107.959545] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1108.977496] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 1111.519891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1111.522375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.526897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.744895] Lustre: server umount lustre-MDT0000 complete [ 1114.869217] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1115.120966] LustreError: 50281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.128831] LustreError: 50281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1117.078313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.122342] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.222395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.241459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1118.245255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1118.496578] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1119.663037] Lustre: DEBUG MARKER: Wait for df (36632) ... [ 1120.172647] Lustre: DEBUG MARKER: done [ 1122.272541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1122.275916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.280831] Lustre: Skipped 2 previous similar messages [ 1122.282802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.284659] Lustre: Skipped 5 previous similar messages [ 1123.208347] Lustre: server umount lustre-OST0000 complete [ 1126.482645] Lustre: server umount lustre-MDT0000 complete [ 1127.687175] LustreError: 50239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904082 with bad export cookie 1984035737796444551 [ 1127.691247] LustreError: 50239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1127.692021] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.188661] Lustre: server umount lustre-MDT0001 complete [ 1141.502399] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1142.409165] Key type lgssc unregistered [ 1142.534343] LNet: 55189:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.536812] LNetError: 55189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.544332] LNet: Removed LNI 192.168.204.115@tcp [ 1142.823375] Key type .llcrypt unregistered [ 1142.824719] Key type ._llcrypt unregistered [ 1156.353756] Key type ._llcrypt registered [ 1156.354778] Key type .llcrypt registered [ 1156.392808] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1156.697668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1156.753520] alg: No test for adler32 (adler32-zlib) [ 1157.595273] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1157.681081] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1159.255141] Key type lgssc registered [ 1159.565464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1161.883736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1161.889768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.963765] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1162.970242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1163.019685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.241014] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1166.932053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.961375] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1166.971304] Lustre: 57372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1167.059849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1168.300028] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1170.435651] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1172.110804] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1174.237018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1174.298639] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1174.358124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.185888] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1178.523176] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1179.662041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1186.806473] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0817h49m01s [ 1187.258956] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1189.328701] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1190.064446] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1190.076972] LustreError: 57378:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1210.413299] Lustre: lustre-MDT0000: Client 4f743ad6-f8fd-4793-9878-cba03eef9898 (at 192.168.204.15@tcp) reconnecting [ 1230.143123] LustreError: 57378:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1231.825153] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1233.554674] Lustre: DEBUG MARKER: done [ 1241.055973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.062032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.066272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.068459] Lustre: Skipped 1 previous similar message [ 1241.080129] Lustre: Skipped 1 previous similar message [ 1242.540532] Lustre: server umount lustre-OST0000 complete [ 1246.179086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.186634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.200790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.202673] Lustre: Skipped 2 previous similar messages [ 1250.850257] Lustre: server umount lustre-MDT0000 complete [ 1252.928563] LustreError: 57364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904207 with bad export cookie 18057050086144820694 [ 1252.929341] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.936512] LustreError: 57364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1253.107945] Lustre: server umount lustre-MDT0001 complete [ 1257.395293] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1258.590908] Key type lgssc unregistered [ 1258.754099] LNet: 61692:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.759942] LNetError: 61692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.769444] LNet: Removed LNI 192.168.204.115@tcp [ 1259.183988] Key type .llcrypt unregistered [ 1259.186353] Key type ._llcrypt unregistered [ 1305.318310] Key type ._llcrypt registered [ 1305.320335] Key type .llcrypt registered [ 1305.402241] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1306.428413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1306.479730] alg: No test for adler32 (adler32-zlib) [ 1307.784597] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1307.988661] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1309.703137] Key type lgssc registered [ 1310.861749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1317.014437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1317.082767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.935757] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1319.079070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1319.660526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.084340] Lustre: Failing over lustre-MDT0000 [ 1321.544234] Lustre: server umount lustre-MDT0000 complete [ 1336.087809] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:51:27 (1788904287) [ 1346.832881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.467255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.024359] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1362.554830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.660517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1362.712464] Lustre: 64497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1363.002539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.555461] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1374.828466] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1380.740465] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1388.075386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1388.309448] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1388.539594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.351741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1397.357641] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1406.317844] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.752541] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1419.236619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1419.252379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.274964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.260239] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.263738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.286554] Lustre: Skipped 1 previous similar message [ 1420.317682] Lustre: Skipped 2 previous similar messages [ 1424.812879] Lustre: server umount lustre-MDT0000 complete [ 1428.467590] LustreError: 64489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904382 with bad export cookie 17491659810212296511 [ 1428.477213] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1428.487430] LustreError: 64489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1428.982887] Lustre: server umount lustre-MDT0001 complete [ 1432.551171] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1437.971461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1445.667682] Lustre: 63417:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904384/real 1788904384] req@000000004ad5ebe6 x1875802075471104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1445.694606] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.770572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.927154] Lustre: 63419:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904387/real 1788904387] req@00000000296ec2c6 x1875802075471232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1451.999215] Lustre: 63418:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904390/real 1788904390] req@00000000c58017e3 x1875802075471360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1452.896730] LustreError: 63415:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007eb1e068 x1875802075472128/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1453.307836] LustreError: 66346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.332563] LustreError: 66346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1453.427138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.527191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1454.433115] LustreError: 66347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.436934] Lustre: 63416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904392/real 1788904392] req@0000000046920c62 x1875802075471616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1456.486575] LustreError: 68265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.519172] LustreError: 68265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1457.932327] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1458.592978] Lustre: 63416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904397/real 1788904397] req@0000000098ed695f x1875802075471744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904413 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1461.729458] LustreError: 68266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.751390] LustreError: 68266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1466.851754] LustreError: 68264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.872035] LustreError: 68264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1467.490060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.810828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1472.446991] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1473.006500] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1475.071582] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1480.199791] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1486.227946] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1492.895992] Lustre: server umount lustre-MDT0000 complete [ 1493.483204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1493.503178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.520840] LustreError: 66347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.618686] LustreError: 64487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904451 with bad export cookie 17491659810212298786 [ 1496.629894] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1496.641877] LustreError: 64487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1499.230789] Lustre: server umount lustre-MDT0001 complete [ 1505.917051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1514.857652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1517.023260] Lustre: 63416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904455/real 1788904455] req@0000000040dadaa2 x1875802075499136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1517.054177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.060450] Lustre: Skipped 2 previous similar messages [ 1522.829196] LustreError: 66346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.861537] LustreError: 66346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1522.985752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.062430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1527.867535] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1536.500187] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1537.590397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.172524] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1544.072802] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1553.092098] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1559.735886] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1567.388977] Lustre: server umount lustre-MDT0000 complete [ 1568.744769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1568.745545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.756309] LustreError: 69804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.756319] LustreError: 69804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 1568.820478] Lustre: Skipped 1 previous similar message [ 1571.529793] LustreError: 64489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904525 with bad export cookie 17491659810212300165 [ 1571.552316] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1571.555607] LustreError: 64489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1571.948740] Lustre: server umount lustre-MDT0001 complete [ 1578.722461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1583.026459] Lustre: DEBUG MARKER: files compared the same [ 1595.708529] Lustre: server umount lustre-OST0000 complete [ 1615.917889] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1618.442836] Key type lgssc unregistered [ 1618.738353] LNet: 74221:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.743237] LNetError: 74221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.757648] LNet: Removed LNI 192.168.204.115@tcp [ 1619.935453] Key type .llcrypt unregistered [ 1619.938957] Key type ._llcrypt unregistered [ 1647.780940] Key type ._llcrypt registered [ 1647.783722] Key type .llcrypt registered [ 1647.882738] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1648.874435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.939834] alg: No test for adler32 (adler32-zlib) [ 1650.033954] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1650.397702] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1652.111255] Key type lgssc registered [ 1653.291954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1662.401477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1662.430333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.037834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.681802] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 1674.209852] LustreError: 75649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1674.225720] LustreError: 75649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1678.131104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.379782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1683.457953] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 1691.431555] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1697.274248] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1705.012442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.276405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.352631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1711.650757] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug +malloc all [ 1720.751751] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1735.140427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1735.152299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.174929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.208675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.211177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.233970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1739.701592] Lustre: server umount lustre-OST0000 complete [ 1744.866959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1744.886407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.907847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.917881] Lustre: Skipped 1 previous similar message [ 1748.449527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.451190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1748.490745] Lustre: Skipped 1 previous similar message [ 1751.086212] Lustre: server umount lustre-MDT0000 complete [ 1753.567843] LustreError: 75650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.576323] LustreError: 75650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1756.593711] LustreError: 75629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904711 with bad export cookie 5320916808842652276 [ 1756.596386] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.609163] LustreError: 75629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1756.920148] Lustre: server umount lustre-MDT0001 complete [ 1770.723872] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1773.825597] Key type lgssc unregistered [ 1774.254161] LNet: 79199:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.268398] LNetError: 79199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.292133] LNet: Removed LNI 192.168.204.115@tcp [ 1775.284491] Key type .llcrypt unregistered [ 1775.286151] Key type ._llcrypt unregistered [ 1794.041036] Key type ._llcrypt registered [ 1794.042916] Key type .llcrypt registered [ 1794.089158] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1794.725739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.746653] alg: No test for adler32 (adler32-zlib) [ 1795.720487] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1795.900265] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1797.551175] Key type lgssc registered [ 1798.320793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1804.045836] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1804.075609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1819.679488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1824.799350] LustreError: 80565:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 1851.423322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.750346] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1858.937313] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1863.020874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.094442] Lustre: *** cfs_fail_loc=706, val=0*** [ 1863.290494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1863.321723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1865.302098] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1869.697901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.941465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1872.192790] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1876.163821] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1879.206414] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1880.545340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.558569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.154594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1885.158457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.165559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.504543] Lustre: server umount lustre-OST0000 complete [ 1890.273373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1890.277143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.286113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.292322] Lustre: Skipped 1 previous similar message [ 1894.634852] Lustre: server umount lustre-MDT0000 complete [ 1895.904446] LustreError: 81654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.916125] LustreError: 81654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1896.799583] LustreError: 80573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904851 with bad export cookie 3268764705423094832 [ 1896.803071] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.807448] LustreError: 80573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1896.979440] Lustre: server umount lustre-MDT0001 complete [ 1902.433296] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 1903.900555] Key type lgssc unregistered [ 1904.057907] LNet: 84173:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.061434] LNetError: 84173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.071623] LNet: Removed LNI 192.168.204.115@tcp [ 1904.467315] Key type .llcrypt unregistered [ 1904.468744] Key type ._llcrypt unregistered [ 1917.537895] Key type ._llcrypt registered [ 1917.539075] Key type .llcrypt registered [ 1917.594274] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 1918.016020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1918.068274] alg: No test for adler32 (adler32-zlib) [ 1918.969255] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1919.100376] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 1920.703231] Key type lgssc registered [ 1921.210155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1924.884711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1924.901616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.802492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1931.518388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.807981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1933.873973] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1936.866434] LustreError: 86266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.886038] LustreError: 86266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1938.284641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1938.494525] LustreError: 86265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.511582] LustreError: 86265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1938.515305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.403750] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1941.600355] LustreError: 86269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.614241] LustreError: 86269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1945.580850] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.589243] Lustre: MGS: Client 8d6d9a7c-ad40-455f-b9f0-34e038182b47 (at 0@lo) reconnecting [ 1945.603909] LustreError: 86280:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1945.704812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.735700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1947.368224] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 1955.808197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1955.812885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.820847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1956.320990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1956.321194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.442027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1961.446588] Lustre: Skipped 1 previous similar message [ 1961.835034] Lustre: server umount lustre-OST0000 complete [ 1966.561852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.561949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1966.569633] Lustre: Skipped 1 previous similar message [ 1966.569917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.569922] Lustre: Skipped 1 previous similar message [ 1971.680964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.690400] Lustre: Skipped 2 previous similar messages [ 1978.335221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1978.391325] Lustre: server umount lustre-MDT0000 complete [ 1980.116222] Lustre: server umount lustre-MDT0001 complete [ 1981.630229] Lustre: server umount MGS complete [ 1986.237249] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:02:20 (1788904940) [ 1997.841365] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2001.437241] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2014.563185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.995039] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2019.970621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.077992] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2020.104074] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2020.143435] Lustre: lustre-MDT0001: new disk, initializing [ 2020.169536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.180149] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2020.182571] Lustre: 93428:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2020.185451] LustreError: 93428:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2021.639940] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2026.037891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2026.135914] Lustre: lustre-OST0000: new disk, initializing [ 2026.138589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2026.141527] Lustre: 94194:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2026.166858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.234949] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2032.096062] Lustre: 94777:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ca38e10c x1875802716146688/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788905002 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2032.277954] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2034.812286] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2034.819599] Lustre: MGS: Client 5bef44d9-2dca-4dbe-a51a-3fa27fc64eb9 (at 0@lo) reconnecting [ 2034.833467] LustreError: 93413:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2034.848554] Lustre: 92698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2034.869238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2034.909595] Lustre: lustre-MDT0000: new disk, initializing [ 2034.979299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.994906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2036.064128] Lustre: 95253:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d9db9c33 x1875802716152064/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788905006 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2036.064604] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2036.601237] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2037.155096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2037.159648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2038.116685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2038.120169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2055.648696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2055.652339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.658268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.660885] Lustre: Skipped 2 previous similar messages [ 2057.700707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2061.650868] Lustre: server umount lustre-OST0000 complete [ 2067.937304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2067.937350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.946858] Lustre: Skipped 1 previous similar message [ 2069.203252] Lustre: server umount lustre-MDT0000 complete [ 2076.773242] Lustre: server umount lustre-MDT0001 complete [ 2078.098079] Lustre: server umount MGS complete [ 2082.344680] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:03:56 (1788905036) [ 2088.932558] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2090.105202] Key type lgssc unregistered [ 2090.238369] LNet: 97755:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.242677] LNetError: 97755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.254374] LNet: Removed LNI 192.168.204.115@tcp [ 2090.605596] Key type .llcrypt unregistered [ 2090.606598] Key type ._llcrypt unregistered [ 2098.730944] Key type ._llcrypt registered [ 2098.732400] Key type .llcrypt registered [ 2098.780810] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2099.269578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.280426] alg: No test for adler32 (adler32-zlib) [ 2100.155761] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2100.256937] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2101.847222] Key type lgssc registered [ 2102.248483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2104.990930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2104.998962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.174457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.631207] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2110.760777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.900593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.322322] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2114.871094] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2116.869356] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2119.312051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2119.435606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.348406] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2123.838725] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2130.726549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2137.055871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2137.058313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.062814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.152964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2141.154518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.154692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.161476] Lustre: Skipped 1 previous similar message [ 2143.122993] Lustre: server umount lustre-OST0000 complete [ 2146.273583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.274698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.277828] Lustre: Skipped 1 previous similar message [ 2150.560749] Lustre: server umount lustre-MDT0000 complete [ 2151.391922] LustreError: 98971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.399163] LustreError: 98971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2151.795458] LustreError: 98933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905106 with bad export cookie 12534335994837565722 [ 2151.798927] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.800990] LustreError: 98933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2151.929715] Lustre: server umount lustre-MDT0001 complete [ 2155.609307] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2156.684627] Key type lgssc unregistered [ 2156.818471] LNet: 102502:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.821384] LNetError: 102502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.829336] LNet: Removed LNI 192.168.204.115@tcp [ 2157.157305] Key type .llcrypt unregistered [ 2157.159218] Key type ._llcrypt unregistered [ 2163.683103] Key type ._llcrypt registered [ 2163.684561] Key type .llcrypt registered [ 2163.725710] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2164.080195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.148210] alg: No test for adler32 (adler32-zlib) [ 2165.003136] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2165.085055] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2166.671179] Key type lgssc registered [ 2167.027573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2169.131993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2169.132098] LustreError: 103562:0:(tgt_mount.c:2659:server_fill_super()) cfs_race id 716 sleeping [ 2169.134712] Lustre: Skipped 1 previous similar message [ 2169.139790] LustreError: 103563:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 waking [ 2169.141877] LustreError: 103562:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2169.144785] LustreError: 103562:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2169.148040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.148975] LustreError: 103562:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2170.310142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.906514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.014208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2174.477548] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2176.300617] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lsmod [ 2177.384672] LustreError: 104767:0:(tgt_mount.c:2659:server_fill_super()) cfs_race id 716 sleeping [ 2177.384746] LustreError: 104768:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 waking [ 2177.391718] LustreError: 104767:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2177.394850] LustreError: 104767:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2177.398211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2177.399845] LustreError: 104767:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2177.540885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.608655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.609600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.613492] Lustre: Skipped 1 previous similar message [ 2182.753507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.754392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.756351] Lustre: Skipped 2 previous similar messages [ 2184.723627] Lustre: server umount lustre-MDT0000 complete [ 2186.120853] LustreError: 103575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905140 with bad export cookie 15576067122982534874 [ 2186.123418] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.125460] LustreError: 103575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2186.847340] LustreError: 104030:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002ccd58c7 x1875802974076928/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2186.910919] Lustre: server umount lustre-MDT0001 complete [ 2198.492757] Lustre: server umount lustre-OST0000 complete [ 2201.480125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.635038] LustreError: 105834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2201.641660] LustreError: 105834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2201.656678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.038411] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2206.034584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.175095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2207.614924] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2210.146644] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2212.221913] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2214.743222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2214.868184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.928249] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2219.534143] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2227.174503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2232.289485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2232.292115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.293718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2232.298118] Lustre: Skipped 1 previous similar message [ 2232.302063] Lustre: Skipped 1 previous similar message [ 2233.685086] Lustre: server umount lustre-OST0000 complete [ 2236.895946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2236.899459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.906442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2237.080710] Lustre: server umount lustre-MDT0000 complete [ 2237.409057] LustreError: 105834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.416290] LustreError: 105834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2238.369379] LustreError: 105816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905192 with bad export cookie 15576067122982535560 [ 2238.372360] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.373737] LustreError: 105816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2238.489068] Lustre: server umount lustre-MDT0001 complete [ 2242.094058] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2243.182422] Key type lgssc unregistered [ 2243.319113] LNet: 109380:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.322601] LNetError: 109380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2243.333327] LNet: Removed LNI 192.168.204.115@tcp [ 2243.643155] Key type .llcrypt unregistered [ 2243.644438] Key type ._llcrypt unregistered [ 2252.876379] Key type ._llcrypt registered [ 2252.877889] Key type .llcrypt registered [ 2252.915681] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2253.230439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2253.240208] alg: No test for adler32 (adler32-zlib) [ 2254.101109] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2254.192862] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2255.775174] Key type lgssc registered [ 2256.201478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2259.080459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2259.087312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.251207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.850893] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2265.103376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.284993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.843654] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2269.570735] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2271.667946] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2274.195971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2274.332909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.432603] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2279.185401] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2284.586079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2291.647556] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2292.453837] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2294.755147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.756075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2294.756824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.760073] Lustre: Skipped 1 previous similar message [ 2299.606876] Lustre: server umount lustre-OST0000 complete [ 2301.408122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2301.410854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.415522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2301.419382] Lustre: Skipped 1 previous similar message [ 2304.992080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.992562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.160381] Lustre: server umount lustre-MDT0000 complete [ 2308.465195] LustreError: 113322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905262 with bad export cookie 7919497858546804979 [ 2308.469047] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.470215] LustreError: 113322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2308.593579] Lustre: server umount lustre-MDT0001 complete [ 2312.138917] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2313.205775] Key type lgssc unregistered [ 2313.334527] LNet: 114425:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2313.337587] LNetError: 114425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2313.345414] LNet: Removed LNI 192.168.204.115@tcp [ 2313.682948] Key type .llcrypt unregistered [ 2313.684401] Key type ._llcrypt unregistered [ 2320.845539] Key type ._llcrypt registered [ 2320.846746] Key type .llcrypt registered [ 2320.885786] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2321.268283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.313950] alg: No test for adler32 (adler32-zlib) [ 2322.176529] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2322.270835] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2323.855160] Key type lgssc registered [ 2324.303166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2327.283381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2327.295621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.477613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.987069] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2333.281352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.421496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2334.965539] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2337.639612] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2339.768434] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2342.310735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2342.456931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.667341] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2347.453565] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2352.678373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2362.850698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2362.853145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.854095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.860276] Lustre: Skipped 1 previous similar message [ 2366.541556] Lustre: server umount lustre-OST0000 complete [ 2367.968332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.968979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2367.972933] Lustre: Skipped 1 previous similar message [ 2367.975492] Lustre: Skipped 1 previous similar message [ 2373.088463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2373.090916] Lustre: Skipped 2 previous similar messages [ 2378.208223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.210874] Lustre: Skipped 1 previous similar message [ 2382.303119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2382.365636] Lustre: server umount lustre-MDT0000 complete [ 2383.329232] LustreError: 116363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.333874] LustreError: 116363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2383.617100] LustreError: 115605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905338 with bad export cookie 18256707831454956612 [ 2383.619753] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.621838] LustreError: 115605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2383.727864] Lustre: server umount lustre-MDT0001 complete [ 2387.323873] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2388.435810] Key type lgssc unregistered [ 2388.572400] LNet: 119177:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.575130] LNetError: 119177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.582304] LNet: Removed LNI 192.168.204.115@tcp [ 2388.923729] Key type .llcrypt unregistered [ 2388.925380] Key type ._llcrypt unregistered [ 2398.279588] Key type ._llcrypt registered [ 2398.281133] Key type .llcrypt registered [ 2398.326175] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2398.687732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2398.732675] alg: No test for adler32 (adler32-zlib) [ 2399.586654] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2399.688532] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2401.271151] Key type lgssc registered [ 2401.642917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2404.382933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2404.388902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.539171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.912330] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2409.940425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.070834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2411.496416] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2414.043702] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2415.883679] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2418.066960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2418.180563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.230046] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2422.850862] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2431.526085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2436.023817] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2437.303697] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2437.307166] Lustre: Skipped 2 previous similar messages [ 2438.726238] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2438.729897] Lustre: Skipped 2 previous similar messages [ 2445.818997] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2445.829950] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2450.855351] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2450.859289] Lustre: Skipped 2 previous similar messages [ 2451.702804] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.15@tcp 192.168.0.[2,10]@tcp [ 2456.714818] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.15/24@tcp in log lustre-MDT0000 [ 2456.717090] Lustre: Skipped 2 previous similar messages [ 2460.600972] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.15/24@tcp [ 2460.603669] Lustre: Skipped 1 previous similar message [ 2466.272585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2466.276837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.283088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.297046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2467.298621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.298872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.307588] Lustre: Skipped 1 previous similar message [ 2467.358527] Lustre: server umount lustre-OST0000 complete [ 2470.742368] Lustre: server umount lustre-MDT0000 complete [ 2471.391727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2471.394185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.398779] LustreError: 120570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.403878] LustreError: 120570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2472.003245] LustreError: 120551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905426 with bad export cookie 6360207068826971235 [ 2472.005889] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.006835] LustreError: 120551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2472.123975] Lustre: server umount lustre-MDT0001 complete [ 2475.736633] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2476.703772] Key type lgssc unregistered [ 2476.828303] LNet: 125616:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2476.830980] LNetError: 125616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.840296] LNet: Removed LNI 192.168.204.115@tcp [ 2477.134211] Key type .llcrypt unregistered [ 2477.135295] Key type ._llcrypt unregistered [ 2487.081028] Key type ._llcrypt registered [ 2487.081915] Key type .llcrypt registered [ 2487.117111] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2487.423928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2487.434506] alg: No test for adler32 (adler32-zlib) [ 2488.287893] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2488.377519] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2489.951145] Key type lgssc registered [ 2490.325074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2492.912948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2492.918651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.062178] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2494.066105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.073303] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.15/24@tcp [ 2495.431502] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2498.230277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.333915] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2498.340616] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.15/24@tcp [ 2498.358747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2499.639166] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2502.062545] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2503.899810] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2506.061680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2506.172679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.944138] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2510.209439] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2520.552700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2525.665344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.666534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2525.666619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2525.671061] Lustre: Skipped 1 previous similar message [ 2529.807128] Lustre: server umount lustre-OST0000 complete [ 2534.367950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2534.370750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.376762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.378642] Lustre: Skipped 1 previous similar message [ 2535.904057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2535.904604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.172985] Lustre: server umount lustre-MDT0000 complete [ 2538.446207] LustreError: 127133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905492 with bad export cookie 12458274846793867900 [ 2538.449932] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.450374] LustreError: 127133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2538.559219] Lustre: server umount lustre-MDT0001 complete [ 2542.126670] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2543.131536] Key type lgssc unregistered [ 2543.271350] LNet: 130760:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.273752] LNetError: 130760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.281290] LNet: Removed LNI 192.168.204.115@tcp [ 2543.571603] Key type .llcrypt unregistered [ 2543.573045] Key type ._llcrypt unregistered [ 2552.083408] Key type ._llcrypt registered [ 2552.084484] Key type .llcrypt registered [ 2552.125808] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2558.357151] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2558.664507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.677392] alg: No test for adler32 (adler32-zlib) [ 2559.532099] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2559.628502] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2561.215135] Key type lgssc registered [ 2561.575584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2573.163251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2573.168921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.251078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2574.262082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2574.298286] Lustre: lustre-MDT0000: new disk, initializing [ 2574.321373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.328288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2575.741648] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2580.064106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.110477] Lustre: 134132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2580.135838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2580.138426] Lustre: Skipped 1 previous similar message [ 2580.173864] Lustre: lustre-MDT0001: new disk, initializing [ 2580.212390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.227065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2580.230464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2581.554499] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2585.400718] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2587.407352] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2589.748764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.855853] Lustre: lustre-OST0000: new disk, initializing [ 2589.858072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2589.861739] Lustre: 136100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2589.891356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.832559] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2595.640328] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2597.110710] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2598.905603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2598.909571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2598.941597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2599.222335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2600.692920] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2600.772099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2604.004508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.004903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.005645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.008400] LustreError: Skipped 1 previous similar message [ 2604.013251] Lustre: Skipped 1 previous similar message [ 2607.577760] Lustre: server umount lustre-OST0000 complete [ 2609.120784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.121422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.125906] Lustre: Skipped 1 previous similar message [ 2609.128317] Lustre: Skipped 1 previous similar message [ 2614.244598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.248948] Lustre: Skipped 1 previous similar message [ 2615.171969] Lustre: server umount lustre-MDT0000 complete [ 2618.291830] LustreError: 134125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905572 with bad export cookie 2484776667214520353 [ 2618.297716] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.302718] LustreError: 134125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2618.582186] Lustre: server umount lustre-MDT0001 complete [ 2626.209233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.683177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.714803] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2632.164682] LustreError: 138333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.196199] LustreError: 138333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2637.286142] LustreError: 138332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.384983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.765943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2645.937819] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2654.786902] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2662.245788] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2671.518745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.981500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.997367] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2683.206705] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2685.718485] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2685.824660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2688.549338] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2688.685418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2693.966505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.065522] Lustre: lustre-OST0001: new disk, initializing [ 2694.067955] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2694.072994] Lustre: 141706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2694.129823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.382713] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2702.376918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2702.382179] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2702.458635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2702.558507] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2702.673722] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2704.840503] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2704.956794] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2708.020544] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff905044e47800.ost_server_uuid 50 [ 2708.824622] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff905044e47800.ost_server_uuid in FULL state after 0 sec [ 2713.057441] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2713.060553] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.067141] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2717.665133] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2717.669989] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2717.670888] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.106304] Lustre: server umount lustre-OST0001 complete [ 2722.272771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2722.279272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.784722] LustreError: 140157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.798893] LustreError: 140157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2727.492394] Lustre: server umount lustre-OST0000 complete [ 2733.026236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.027433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2733.040458] Lustre: Skipped 1 previous similar message [ 2733.055752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2733.059292] Lustre: Skipped 4 previous similar messages [ 2735.864170] Lustre: server umount lustre-MDT0000 complete [ 2737.998707] LustreError: 138313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905692 with bad export cookie 2484776667214521221 [ 2738.000640] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2738.005274] LustreError: 138313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2738.181912] Lustre: server umount lustre-MDT0001 complete [ 2748.444090] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2749.849836] Key type lgssc unregistered [ 2750.002827] LNet: 144704:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2750.007451] LNetError: 144704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2750.018859] LNet: Removed LNI 192.168.204.115@tcp [ 2750.476912] Key type .llcrypt unregistered [ 2750.479258] Key type ._llcrypt unregistered [ 2771.064837] Key type ._llcrypt registered [ 2771.066854] Key type .llcrypt registered [ 2771.140358] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 2779.964570] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 2780.574976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2780.590290] alg: No test for adler32 (adler32-zlib) [ 2781.492762] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2781.674123] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 2783.295164] Key type lgssc registered [ 2783.958575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2804.761460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2804.775634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.934769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2805.954534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2806.011629] Lustre: lustre-MDT0000: new disk, initializing [ 2806.066958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.081712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2808.705507] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2815.570118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.647931] Lustre: 148831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2815.680093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2815.683250] Lustre: Skipped 1 previous similar message [ 2815.735307] Lustre: lustre-MDT0001: new disk, initializing [ 2815.803854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2815.826640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2815.833447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2818.294817] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2824.392322] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2827.890400] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2832.442593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2832.666265] Lustre: lustre-OST0000: new disk, initializing [ 2832.671832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2832.682134] Lustre: 150799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2832.750110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.736330] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2842.145885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2842.153978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2842.220345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2843.018626] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2847.028632] Lustre: Failing over lustre-OST0000 [ 2847.200166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2847.201831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.209299] LustreError: Skipped 1 previous similar message [ 2847.219507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.224289] Lustre: Skipped 1 previous similar message [ 2849.158150] Lustre: server umount lustre-OST0000 complete [ 2865.191172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2865.387217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.405422] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2866.177471] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2867.450956] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2867.451728] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2868.968280] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2871.308820] Lustre: Failing over lustre-MDT0000 [ 2871.483905] Lustre: server umount lustre-MDT0000 complete [ 2872.289671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2872.293840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.302488] Lustre: Skipped 1 previous similar message [ 2877.922608] LustreError: 148844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.941703] LustreError: 148844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2881.522868] LustreError: 148839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.040841] LustreError: 148840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.052756] LustreError: 148840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2886.644748] LustreError: 148839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.759635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.853760] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2886.860650] Lustre: 148835:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.115@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2887.033256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.074580] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2888.719206] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2889.600829] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 2892.265150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2892.271586] Lustre: Skipped 1 previous similar message [ 2892.289528] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2892.324849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2897.383171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.385639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.391790] Lustre: Skipped 3 previous similar messages [ 2897.403688] Lustre: Skipped 1 previous similar message [ 2900.708344] Lustre: server umount lustre-OST0000 complete [ 2907.620428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2907.621091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.633084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.640659] Lustre: Skipped 1 previous similar message [ 2912.740127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2912.749523] Lustre: Skipped 1 previous similar message [ 2916.832895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2916.840929] Lustre: Skipped 1 previous similar message [ 2917.343231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2917.480572] Lustre: server umount lustre-MDT0000 complete [ 2921.954208] LustreError: 148844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.982346] LustreError: 148844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2923.076226] LustreError: 151604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905877 with bad export cookie 8745391115129746771 [ 2923.079483] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.113140] LustreError: 151604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2924.539431] Lustre: server umount lustre-MDT0001 complete [ 2943.728871] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 2948.398551] Key type lgssc unregistered [ 2948.796701] LNet: 154472:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2948.808582] LNetError: 154472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2948.830314] LNet: Removed LNI 192.168.204.115@tcp [ 2949.906660] Key type .llcrypt unregistered [ 2949.908453] Key type ._llcrypt unregistered [ 3001.646710] Key type ._llcrypt registered [ 3001.653786] Key type .llcrypt registered [ 3001.888054] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3003.533844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3003.563584] alg: No test for adler32 (adler32-zlib) [ 3004.641365] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3004.944168] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 3006.655205] Key type lgssc registered [ 3008.218193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3016.817364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3016.834340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.365307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.514792] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3027.937594] LustreError: 155997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.974880] LustreError: 155997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3033.059348] LustreError: 155996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.978831] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3038.177100] LustreError: 155997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.273058] LustreError: 156025:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3038.284981] LustreError: 156025:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3038.297688] LustreError: 156025:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 3038.662427] Lustre: server umount lustre-MDT0000 complete [ 3054.687920] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3065.776760] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3105.368288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.676442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3105.722488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3105.819214] Lustre: lustre-MDT0000: new disk, initializing [ 3105.882643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.902469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3110.676649] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3121.882994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.051449] Lustre: 159327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3122.117475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3122.121850] Lustre: Skipped 1 previous similar message [ 3122.194234] Lustre: lustre-MDT0001: new disk, initializing [ 3122.309868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3122.368136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3122.388047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3126.684929] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3137.694895] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3144.537112] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3151.472559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.851373] Lustre: lustre-OST0000: new disk, initializing [ 3151.860571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3151.877935] Lustre: 161297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3152.013398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.516541] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3161.689973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3161.710222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3161.879249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3168.694894] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3174.195830] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3176.195735] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:22:09 (1788906129) [ 3448.804194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.805143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.825913] Lustre: Skipped 2 previous similar messages [ 3448.833507] Lustre: Skipped 1 previous similar message [ 3453.925724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3453.944799] Lustre: Skipped 3 previous similar messages [ 3454.471821] Lustre: server umount lustre-MDT0000 complete [ 3458.031885] LustreError: 163062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906412 with bad export cookie 10524589752112338803 [ 3458.032674] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3458.042572] LustreError: 163062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3459.040462] LustreError: 159335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.047215] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.071110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3463.071652] LustreError: 159337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3463.071813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3468.192191] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3472.351561] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3472.709592] Lustre: server umount lustre-MDT0001 complete [ 3485.509677] Lustre: server umount lustre-OST0000 complete [ 3499.130955] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3512.733292] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3556.576753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.925974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3557.006746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3557.170165] Lustre: lustre-MDT0000: new disk, initializing [ 3557.347410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.384904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3562.471279] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3574.468717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.614804] Lustre: 165977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3574.667920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3574.673909] Lustre: Skipped 1 previous similar message [ 3574.772912] Lustre: lustre-MDT0001: new disk, initializing [ 3574.896858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3574.944362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3574.967289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3581.099505] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3592.135841] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3598.106988] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3605.832717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.047619] Lustre: lustre-OST0000: new disk, initializing [ 3606.051778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3606.067649] Lustre: 167945:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3606.197423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.745185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3612.760333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3612.895553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3613.662334] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3625.337516] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3630.410220] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3630.618126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3636.095370] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3636.305716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3638.753568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.759684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3638.775923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3644.705777] Lustre: server umount lustre-OST0000 complete [ 3648.992636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.020034] Lustre: Skipped 2 previous similar messages [ 3659.232455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.248224] Lustre: Skipped 8 previous similar messages [ 3663.331816] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3663.678326] Lustre: server umount lustre-MDT0000 complete [ 3664.358777] LustreError: 167287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3664.407918] LustreError: 167287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3668.228266] LustreError: 166771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906622 with bad export cookie 10524589752112497640 [ 3668.246709] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.248895] LustreError: 166771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3668.889064] Lustre: server umount lustre-MDT0001 complete [ 3677.961400] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:30:31 (1788906631) [ 3693.211422] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3702.694713] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3734.324611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3734.555985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3734.587140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3734.703691] Lustre: lustre-MDT0000: new disk, initializing [ 3734.800762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.830394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3739.315823] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3748.595761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.781297] Lustre: 172708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3749.096545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3754.047656] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3762.762570] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3768.279503] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3775.687174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3776.083384] Lustre: lustre-OST0000: new disk, initializing [ 3776.085855] Lustre: Skipped 1 previous similar message [ 3776.088857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3776.092317] Lustre: Skipped 2 previous similar messages [ 3776.099239] Lustre: 174685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3782.217734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3782.227903] Lustre: Skipped 1 previous similar message [ 3782.232886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3782.346051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3782.675495] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3791.522440] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3803.118762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.120913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.137178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.166447] Lustre: Skipped 1 previous similar message [ 3813.343143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3813.576640] Lustre: server umount lustre-OST0000 complete [ 3818.464401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.473906] Lustre: Skipped 1 previous similar message [ 3823.894977] Lustre: server umount lustre-MDT0000 complete [ 3828.121344] LustreError: 172700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906782 with bad export cookie 10524589752112498515 [ 3828.131461] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.138538] LustreError: 172700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3828.518746] Lustre: server umount lustre-MDT0001 complete [ 3836.848413] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:33:10 (1788906790) [ 3851.806894] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 3860.747151] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 3896.228240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.450528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3896.474308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3896.539843] Lustre: lustre-MDT0000: new disk, initializing [ 3896.601849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.606575] Lustre: Skipped 2 previous similar messages [ 3896.618513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3901.322591] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3911.487265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.644182] Lustre: 179007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3911.804546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3916.309160] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3925.781517] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3931.830129] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3938.701217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3938.963871] Lustre: 180988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3944.511582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3944.613171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3944.692335] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 3953.987896] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3965.416164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3965.431726] LustreError: Skipped 1 previous similar message [ 3965.437415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.451341] Lustre: Skipped 1 previous similar message [ 3965.456641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3965.463433] Lustre: Skipped 8 previous similar messages [ 3967.453268] Lustre: server umount lustre-OST0000 complete [ 3973.090066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3973.102634] LustreError: Skipped 1 previous similar message [ 3985.375871] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3985.593438] Lustre: server umount lustre-MDT0000 complete [ 3988.701792] LustreError: 180987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906943 with bad export cookie 10524589752112499607 [ 3988.704340] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.712423] LustreError: 180987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3988.959749] Lustre: server umount lustre-MDT0001 complete [ 3998.029556] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4000.002552] Key type lgssc unregistered [ 4000.244706] LNet: 182842:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4000.254844] LNetError: 182842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4001.318736] LNet: Removed LNI 192.168.204.115@tcp [ 4001.940184] Key type .llcrypt unregistered [ 4001.942227] Key type ._llcrypt unregistered [ 4022.000918] Key type ._llcrypt registered [ 4022.002549] Key type .llcrypt registered [ 4022.090664] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4023.000407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4023.068829] alg: No test for adler32 (adler32-zlib) [ 4024.075660] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4024.284621] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4025.959166] Key type lgssc registered [ 4026.843704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4033.195646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4033.210493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4034.530522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.127667] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4043.748747] LustreError: 184250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.773306] LustreError: 184250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4045.367424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.621830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4049.343766] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4056.106537] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4061.062919] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4066.413569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4066.768201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.240716] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4076.625448] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4079.101248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4099.554751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4099.557032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.563717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.573724] Lustre: Skipped 1 previous similar message [ 4099.582072] Lustre: Skipped 1 previous similar message [ 4103.475430] Lustre: server umount lustre-OST0000 complete [ 4107.236671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4107.243659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4107.262853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.793555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.801334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.773571] Lustre: server umount lustre-MDT0000 complete [ 4114.913817] LustreError: 184246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.942742] LustreError: 184246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4116.069581] LustreError: 184231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907070 with bad export cookie 7524966542344408693 [ 4116.077432] LustreError: 184231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4116.078301] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4116.411513] Lustre: server umount lustre-MDT0001 complete [ 4124.388326] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 4126.482425] Key type lgssc unregistered [ 4126.784862] LNet: 187808:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4126.789899] LNetError: 187808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4126.816154] LNet: Removed LNI 192.168.204.115@tcp [ 4127.517161] Key type .llcrypt unregistered [ 4127.518938] Key type ._llcrypt unregistered [ 4143.542791] Key type ._llcrypt registered [ 4143.544872] Key type .llcrypt registered [ 4143.617737] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4144.341350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4144.354674] alg: No test for adler32 (adler32-zlib) [ 4145.280121] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4145.463952] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 4147.095228] Key type lgssc registered [ 4147.974933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4153.522338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4153.543984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.931482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.772167] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4164.003982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.319866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.306546] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4172.685982] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4176.950239] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4182.282898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.537108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.780627] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4194.029438] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4199.987735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4205.024507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4205.030758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.034840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4205.050606] Lustre: Skipped 1 previous similar message [ 4206.090813] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 4206.096667] Lustre: Skipped 1 previous similar message [ 4210.151384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4210.587378] Lustre: server umount lustre-OST0000 complete [ 4217.199907] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff905050648800.ost_server_uuid 50 [ 4218.687611] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4235.360980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.367447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.379093] Lustre: Skipped 1 previous similar message [ 4235.392606] Lustre: Skipped 2 previous similar messages [ 4237.447794] Lustre: server umount lustre-MDT0000 complete [ 4240.578803] LustreError: 189184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907195 with bad export cookie 5734769106061882927 [ 4240.589272] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4240.607744] LustreError: 189184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4240.842346] Lustre: server umount lustre-MDT0001 complete [ 4247.109645] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:40:00 (1788907200) [ 4252.329708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.670919] LustreError: 193117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.688923] LustreError: 193117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4252.734673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.000617] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4257.761696] LustreError: 193118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.890882] LustreError: 193117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.982318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.300656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4266.683320] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4272.416155] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4277.028614] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4282.674804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.945868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.684344] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4293.228582] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4297.806403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4297.981386] Lustre: lustre-OST0001: new disk, initializing [ 4297.987381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4297.999551] Lustre: 195954:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4299.121496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4302.256935] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4307.006576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4307.018987] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4307.118970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4309.562742] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4312.555739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.573508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.580627] Lustre: Skipped 1 previous similar message [ 4317.836639] Lustre: server umount lustre-OST0000 complete [ 4320.263624] LustreError: 194937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.379894] LustreError: 194938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.410558] LustreError: 194938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4327.455549] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4327.902554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4333.592854] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4333.973900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4335.613274] LustreError: 194936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.640942] LustreError: 194936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4352.430888] Lustre: server umount lustre-OST0001 complete [ 4358.626342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.660925] Lustre: Skipped 1 previous similar message [ 4358.683692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.697390] Lustre: Skipped 4 previous similar messages [ 4366.974293] Lustre: server umount lustre-MDT0000 complete [ 4368.863395] Lustre: 188513:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788907307/real 1788907307] req@000000004bbc94e6 x1875805050331776/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788907323 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4368.864403] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.901030] Lustre: 188513:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4368.901377] LustreError: 193112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.901384] LustreError: 193112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4369.000133] Lustre: Skipped 1 previous similar message [ 4372.126369] LustreError: 193099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907326 with bad export cookie 5734769106061883998 [ 4372.144829] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.147413] LustreError: 193099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4372.649361] Lustre: server umount lustre-MDT0001 complete [ 4401.125419] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:42:34 (1788907354) [ 4408.621588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.942506] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4408.991055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4409.254473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.261996] Lustre: Skipped 1 previous similar message [ 4413.446693] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4421.263661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.360458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4421.410647] Lustre: 199397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4425.576662] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4432.051582] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4437.268136] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4443.503878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4443.724297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4443.957385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4443.964241] Lustre: Skipped 1 previous similar message [ 4449.355227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4449.868509] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4456.597151] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4462.675599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4462.742416] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4469.019415] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4477.295360] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4483.100548] Lustre: lustre-OST0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 4483.104628] Lustre: Skipped 2 previous similar messages [ 4484.086900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.544699] Lustre: server umount lustre-OST0000 complete [ 4488.189667] LustreError: 201247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4488.210254] LustreError: 201247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4504.550235] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4504.563940] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.583429] Lustre: Skipped 2 previous similar messages [ 4509.320526] Lustre: server umount lustre-OST0001 complete [ 4513.764547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4513.772654] LustreError: Skipped 1 previous similar message [ 4519.791175] Lustre: server umount lustre-MDT0000 complete [ 4523.835686] LustreError: 203039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907478 with bad export cookie 5734769106061885405 [ 4523.839304] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4523.848017] LustreError: 203039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4524.357520] Lustre: server umount lustre-MDT0001 complete [ 4547.279741] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:45:00 (1788907500) [ 4562.766088] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 4572.382333] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 4606.765211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.017047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4607.045679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4607.129360] Lustre: lustre-MDT0000: new disk, initializing [ 4607.231693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.240211] Lustre: Skipped 1 previous similar message [ 4607.249251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4612.059744] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4621.791628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.961056] Lustre: 207491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4622.040963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4622.045587] Lustre: Skipped 1 previous similar message [ 4622.120664] Lustre: lustre-MDT0001: new disk, initializing [ 4622.224354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4622.232636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4626.509321] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4636.410252] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4642.642016] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4649.642310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4649.988397] Lustre: lustre-OST0000: new disk, initializing [ 4649.993359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4650.001324] Lustre: 209459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4656.022733] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4659.736164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4659.743685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4659.887259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4665.812878] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4670.149413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4670.407983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4674.879550] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4675.108381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4680.166839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4680.174465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.194970] Lustre: Skipped 2 previous similar messages [ 4680.204045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.213696] Lustre: Skipped 7 previous similar messages [ 4683.213416] Lustre: server umount lustre-OST0000 complete [ 4688.866322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4688.875327] LustreError: Skipped 1 previous similar message [ 4701.151191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4701.310167] Lustre: server umount lustre-MDT0000 complete [ 4704.588628] LustreError: 207482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907659 with bad export cookie 5734769106061886966 [ 4704.600617] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4704.605347] LustreError: 207482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4704.882211] Lustre: server umount lustre-MDT0001 complete [ 4711.925074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.291990] LustreError: 211676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.314396] LustreError: 211676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 4716.052875] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4722.857162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.710756] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4732.684582] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4736.956711] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4741.754886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4741.960308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4741.964770] Lustre: Skipped 4 previous similar messages [ 4746.654640] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4752.177400] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4755.215748] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4757.502414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4760.371981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4760.535868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4762.595072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4762.598478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.601065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4762.601073] Lustre: Skipped 8 previous similar messages [ 4762.619732] Lustre: Skipped 4 previous similar messages [ 4764.027509] Lustre: server umount lustre-OST0000 complete [ 4768.341830] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4771.569668] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4798.059769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.874369] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4803.371347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.379643] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.379860] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4803.394508] Lustre: Skipped 1 previous similar message [ 4806.893937] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4818.470888] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 4824.035269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4824.043736] LustreError: Skipped 1 previous similar message [ 4826.982808] Lustre: server umount lustre-OST0000 complete [ 4835.574670] Lustre: server umount lustre-MDT0000 complete [ 4837.879207] LustreError: 211657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907792 with bad export cookie 5734769106061887848 [ 4837.880919] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4837.886940] LustreError: 211657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4838.124906] Lustre: server umount lustre-MDT0001 complete [ 4843.800090] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:49:57 (1788907797) [ 4848.443267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.852196] LustreError: 217677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.869356] LustreError: 217677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4851.937800] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4858.733290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.180637] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4866.706634] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4870.522143] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4875.338309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4879.881484] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4885.425290] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4889.271669] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4895.742498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 4899.309477] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4899.508914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4904.775949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.939223] Lustre: lustre-OST0001: new disk, initializing [ 4904.945493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4904.951898] Lustre: 221086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4909.067502] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4910.154134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4910.162288] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4910.247176] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4916.164398] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4919.537633] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4919.708404] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4922.937690] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4923.121259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4925.932037] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.932838] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4925.944552] Lustre: Skipped 4 previous similar messages [ 4925.950875] LustreError: Skipped 1 previous similar message [ 4925.955701] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4925.961349] Lustre: Skipped 7 previous similar messages [ 4930.415732] Lustre: server umount lustre-OST0001 complete [ 4935.120711] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4935.301443] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4938.747464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4938.915829] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4965.862739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4971.241730] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 4971.497080] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4971.509770] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4971.510275] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4971.526115] Lustre: Skipped 1 previous similar message [ 4977.254965] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4978.460273] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4986.004715] Lustre: server umount lustre-OST0001 complete [ 5001.697387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5001.701557] LustreError: Skipped 2 previous similar messages [ 5006.260112] Lustre: server umount lustre-OST0000 complete [ 5014.805884] Lustre: server umount lustre-MDT0000 complete [ 5017.429854] LustreError: 217658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907971 with bad export cookie 5734769106061888814 [ 5017.431120] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5017.437112] LustreError: 217658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5023.750694] Lustre: server umount lustre-MDT0001 complete [ 5039.144884] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:53:12 (1788907992) [ 5043.384131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.580406] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5043.609129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5043.772606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5043.778552] Lustre: Skipped 6 previous similar messages [ 5046.406391] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5052.100422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.196483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5052.231230] Lustre: 226322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5055.066808] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5059.678721] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5063.166875] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5067.218737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5067.348179] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5071.114779] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5075.905803] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5082.885548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5082.953834] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5087.389670] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5092.357259] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5093.548475] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5096.496501] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5096.647715] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5099.139856] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5099.284845] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5102.811868] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff905046263800.ost_server_uuid 50 [ 5103.853326] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff905046263800.ost_server_uuid in FULL state after 0 sec [ 5104.543134] Lustre: Permanently deactivating lustre-OST0001 [ 5104.554756] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5106.644443] Lustre: Permanently reactivating lustre-OST0001 [ 5111.128300] Lustre: server umount lustre-OST0001 complete [ 5113.315872] LustreError: 228170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.326740] LustreError: 228170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 30 previous similar messages [ 5119.740202] Lustre: server umount lustre-OST0000 complete [ 5136.351214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5136.480381] Lustre: server umount lustre-MDT0000 complete [ 5139.238078] LustreError: 226317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908093 with bad export cookie 5734769106061890102 [ 5139.246400] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.249816] LustreError: 226317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5139.463126] Lustre: server umount lustre-MDT0001 complete [ 5155.637794] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:55:09 (1788908109) [ 5161.306097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.591258] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5161.612389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5161.619548] Lustre: Skipped 5 previous similar messages [ 5165.189748] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5171.565731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5171.623157] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5171.639114] Lustre: 233011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5175.030962] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5180.346617] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5184.175476] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5189.240606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5189.429888] Lustre: Permanently deactivating lustre-OST0000 [ 5189.439609] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5193.793611] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5198.860889] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5198.915662] LustreError: 235514:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 5198.919808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5199.077131] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5203.491599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.721376] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5207.192440] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5211.536505] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5212.814246] Lustre: Permanently reactivating lustre-OST0000 [ 5212.826953] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5212.832542] Lustre: Skipped 2 previous similar messages [ 5214.069835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.089263] Lustre: Skipped 13 previous similar messages [ 5214.101097] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5214.109857] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.118800] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5214.152978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.172413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5223.916438] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 5223.923021] LustreError: Skipped 4 previous similar messages [ 5223.931052] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5223.938246] Lustre: Skipped 21 previous similar messages [ 5229.745709] Lustre: server umount lustre-OST0001 complete [ 5238.415325] Lustre: server umount lustre-OST0000 complete [ 5247.323081] Lustre: server umount lustre-MDT0000 complete [ 5250.078714] LustreError: 233003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908204 with bad export cookie 5734769106061891740 [ 5250.080523] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5250.088552] LustreError: 233003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5250.320532] Lustre: server umount lustre-MDT0001 complete [ 5257.432675] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5259.260826] Key type lgssc unregistered [ 5259.475480] LNet: 237863:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5259.483416] LNetError: 237863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5259.499079] LNet: Removed LNI 192.168.204.115@tcp [ 5260.071983] Key type .llcrypt unregistered [ 5260.074964] Key type ._llcrypt unregistered [ 5273.039404] Key type ._llcrypt registered [ 5273.042155] Key type .llcrypt registered [ 5273.109210] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5273.742748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5273.765375] alg: No test for adler32 (adler32-zlib) [ 5274.716545] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5274.868833] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5276.479322] Key type lgssc registered [ 5277.147154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5284.033427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5284.053344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5285.414078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.581783] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5295.238953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.356928] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5295.385852] Lustre: 239297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5295.394195] Lustre: Permanently deactivating lustre-MDT0001 [ 5295.414202] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5295.648962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5298.663930] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5299.087627] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5303.833438] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5307.650288] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5312.092967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5312.329828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5316.869684] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5320.565721] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5321.573944] LustreError: 241149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5321.586800] LustreError: 241149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5322.429391] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5326.819439] LustreError: 241151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.758932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5327.920893] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5332.025216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5332.030522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5332.034640] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5333.270790] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5338.459841] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5341.809827] Lustre: Permanently reactivating lustre-MDT0001 [ 5341.825443] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5341.831799] Lustre: Skipped 2 previous similar messages [ 5342.502653] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.511661] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 5342.518486] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5342.526905] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5344.791136] Lustre: lustre-MDT0001: Client 2d332c14-f8e7-4ef9-94a1-5723a4343442 (at 192.168.204.15@tcp) reconnecting [ 5348.847175] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5349.982887] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5354.240655] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5354.408826] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5357.605731] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5357.757821] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5359.673468] Lustre: Permanently deactivating lustre-MDT0001 [ 5359.690186] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5359.694687] Lustre: Skipped 2 previous similar messages [ 5367.890113] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5372.384892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5372.393899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.409329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5377.983644] Lustre: server umount lustre-MDT0000 complete [ 5378.020995] LustreError: 239310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.033068] LustreError: 239310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5380.259278] LustreError: 242855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908334 with bad export cookie 8467958447671973009 [ 5380.260109] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5380.267075] LustreError: 242855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5380.467834] Lustre: server umount lustre-MDT0001 complete [ 5389.017730] Lustre: server umount lustre-OST0000 complete [ 5395.412387] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:59:09 (1788908349) [ 5401.906958] Lustre: server umount lustre-OST0001 complete [ 5405.749486] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 5411.691069] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5433.219040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.394792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5433.403560] Lustre: Skipped 2 previous similar messages [ 5433.422484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5433.471203] Lustre: lustre-MDT0000: new disk, initializing [ 5433.528248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5433.544666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5436.486760] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5443.824783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.908729] Lustre: 247557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5443.957370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5443.961957] Lustre: Skipped 1 previous similar message [ 5444.020650] Lustre: lustre-MDT0001: new disk, initializing [ 5444.081864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5444.106520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5444.117461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5446.832526] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5452.168332] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5454.806884] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5458.518600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5458.683883] Lustre: lustre-OST0000: new disk, initializing [ 5458.688548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5458.696360] Lustre: 249525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5462.100696] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5467.142745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5467.149759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5467.197462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5467.595350] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5468.593074] LustreError: 247566:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5473.977628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5474.051099] Lustre: lustre-OST0001: new disk, initializing [ 5474.054245] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5474.059859] Lustre: 250640:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5474.101222] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5474.106601] Lustre: Skipped 1 previous similar message [ 5475.235157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5475.240238] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5475.312430] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5477.280543] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5478.615172] LustreError: 247566:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5478.633782] LustreError: 247565:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5483.184848] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5484.391853] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5488.735172] LustreError: 247565:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5490.192041] Lustre: lustre-OST0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 5490.199904] Lustre: Skipped 3 previous similar messages [ 5490.662461] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5490.663440] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5490.668561] LustreError: Skipped 1 previous similar message [ 5490.679793] Lustre: Skipped 4 previous similar messages [ 5495.284717] Lustre: lustre-OST0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 5495.291361] Lustre: Skipped 2 previous similar messages [ 5496.081543] Lustre: server umount lustre-OST0001 complete [ 5500.902577] LustreError: 249509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.911988] LustreError: 249509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5506.019596] LustreError: 250198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.032352] LustreError: 250198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5511.149318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.158909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5511.164265] Lustre: Skipped 3 previous similar messages [ 5515.765107] Lustre: server umount lustre-OST0000 complete [ 5520.866384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5520.872712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.887256] Lustre: Skipped 1 previous similar message [ 5520.894142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5524.233694] Lustre: server umount lustre-MDT0000 complete [ 5526.405257] LustreError: 247550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908480 with bad export cookie 8467958447671975830 [ 5526.407523] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5526.416101] LustreError: 247550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5532.764756] Lustre: server umount lustre-MDT0001 complete [ 5539.476817] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5541.010796] Key type lgssc unregistered [ 5541.208449] LNet: 252680:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5541.212972] LNetError: 252680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5541.228508] LNet: Removed LNI 192.168.204.115@tcp [ 5541.728092] Key type .llcrypt unregistered [ 5541.729669] Key type ._llcrypt unregistered [ 5563.885248] Key type ._llcrypt registered [ 5563.887777] Key type .llcrypt registered [ 5563.955018] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5564.491777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5564.564757] alg: No test for adler32 (adler32-zlib) [ 5565.507456] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5565.672382] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5567.303183] Key type lgssc registered [ 5567.941319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5572.599873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5572.622210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.777266] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5573.792192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5573.925435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.809237] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5582.304666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.424416] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5582.461093] Lustre: 254864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5582.715464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5585.464499] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5590.124600] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5593.401626] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5597.543709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5597.690763] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5597.855140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.785464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5606.777517] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5607.117816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5663.719358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5663.736729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.761255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5668.321165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5668.336913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.365929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.062193] Lustre: server umount lustre-OST0000 complete [ 5675.001381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5690.567396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5691.008754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.076992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.097554] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5698.371894] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5707.159400] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5709.485763] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.509375] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5750.772239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5750.784077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5750.802133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5750.810504] Lustre: Skipped 1 previous similar message [ 5753.844963] Lustre: server umount lustre-OST0000 complete [ 5760.997315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.009691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5761.022106] Lustre: Skipped 2 previous similar messages [ 5761.039672] Lustre: Skipped 2 previous similar messages [ 5764.414869] Lustre: server umount lustre-MDT0000 complete [ 5768.572469] LustreError: 254857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908722 with bad export cookie 8968612940432943628 [ 5768.588560] LustreError: 254857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5768.590754] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.096558] Lustre: server umount lustre-MDT0001 complete [ 5780.707721] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5783.600601] Key type lgssc unregistered [ 5783.874202] LNet: 260285:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5783.878795] LNetError: 260285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5783.894747] LNet: Removed LNI 192.168.204.115@tcp [ 5784.720551] Key type .llcrypt unregistered [ 5784.726373] Key type ._llcrypt unregistered [ 5810.616655] Key type ._llcrypt registered [ 5810.618739] Key type .llcrypt registered [ 5810.721431] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5811.507548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5811.525833] alg: No test for adler32 (adler32-zlib) [ 5812.570645] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5812.905850] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5814.599193] Key type lgssc registered [ 5815.842886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5823.355110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5823.375409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.032380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5829.931926] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5835.243646] LustreError: 261702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5835.276050] LustreError: 261702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5839.374817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.863992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5844.165158] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5851.806370] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5857.801432] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5864.863499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5865.174966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.338827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5871.331777] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5879.638872] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5906.412550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5906.433167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5906.437491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5906.453234] Lustre: Skipped 1 previous similar message [ 5910.725270] Lustre: server umount lustre-OST0000 complete [ 5916.642676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.642735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5916.653050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5916.664168] Lustre: Skipped 2 previous similar messages [ 5920.607838] Lustre: server umount lustre-MDT0000 complete [ 5921.768605] LustreError: 261701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5921.791881] LustreError: 261701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5924.723409] LustreError: 261681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908879 with bad export cookie 1959039681693984627 [ 5924.732459] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.738886] LustreError: 261681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5925.224411] Lustre: server umount lustre-MDT0001 complete [ 5936.735524] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 5939.190793] Key type lgssc unregistered [ 5939.448568] LNet: 266008:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5939.457424] LNetError: 266008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5939.497827] LNet: Removed LNI 192.168.204.115@tcp [ 5940.314803] Key type .llcrypt unregistered [ 5940.319722] Key type ._llcrypt unregistered [ 5956.940216] Key type ._llcrypt registered [ 5956.942555] Key type .llcrypt registered [ 5957.038734] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 5958.002293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5958.024608] alg: No test for adler32 (adler32-zlib) [ 5959.029512] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5959.257753] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 5960.927284] Key type lgssc registered [ 5962.130691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5969.616370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5969.657355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.279376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5976.828638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 5980.646447] LustreError: 267236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5980.670669] LustreError: 267236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5985.762026] LustreError: 267235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5986.349174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.902255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5992.088723] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6000.561649] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6007.370948] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6014.997446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6015.425834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.438941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6022.367089] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6029.993617] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6042.095451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6042.106863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6042.111466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6042.115668] Lustre: Skipped 1 previous similar message [ 6042.148102] Lustre: Skipped 1 previous similar message [ 6047.204426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6047.525716] Lustre: server umount lustre-OST0000 complete [ 6052.321233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.340268] Lustre: Skipped 1 previous similar message [ 6052.341742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6052.358116] Lustre: Skipped 2 previous similar messages [ 6057.441926] LustreError: 267232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.458866] LustreError: 267232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6057.500368] Lustre: server umount lustre-MDT0000 complete [ 6061.582257] LustreError: 269959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909016 with bad export cookie 6210833030828658172 [ 6061.584892] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6061.595949] LustreError: 269959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6061.932890] Lustre: server umount lustre-MDT0001 complete [ 6073.589492] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6076.220464] Key type lgssc unregistered [ 6076.483580] LNet: 270912:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6076.494726] LNetError: 270912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6076.517892] LNet: Removed LNI 192.168.204.115@tcp [ 6077.273121] Key type .llcrypt unregistered [ 6077.275795] Key type ._llcrypt unregistered [ 6101.462886] Key type ._llcrypt registered [ 6101.464813] Key type .llcrypt registered [ 6101.586171] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6102.544239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6102.698099] alg: No test for adler32 (adler32-zlib) [ 6103.735550] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6103.972420] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6105.671175] Key type lgssc registered [ 6106.743230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6113.926081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6113.948853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6115.352917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6119.142513] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6125.538194] LustreError: 272328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.547743] LustreError: 272328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6126.353513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.615721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6129.932831] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6136.703821] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6142.755951] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6149.360670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6149.600589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6155.605725] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6159.872899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6162.815667] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6185.440715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.448255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.474478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6190.563668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6190.576570] Lustre: Skipped 2 previous similar messages [ 6190.741246] Lustre: server umount lustre-OST0000 complete [ 6195.684402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.687451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.697271] Lustre: Skipped 2 previous similar messages [ 6195.710449] Lustre: Skipped 1 previous similar message [ 6200.537443] Lustre: server umount lustre-MDT0000 complete [ 6200.806611] LustreError: 273056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.822600] LustreError: 273056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6204.091722] LustreError: 272308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909158 with bad export cookie 13600651648538740803 [ 6204.097188] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6204.101346] LustreError: 272308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6204.412652] Lustre: server umount lustre-MDT0001 complete [ 6214.696096] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6217.439492] Key type lgssc unregistered [ 6217.717591] LNet: 276631:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6217.726199] LNetError: 276631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6218.790664] LNet: Removed LNI 192.168.204.115@tcp [ 6219.499214] Key type .llcrypt unregistered [ 6219.504311] Key type ._llcrypt unregistered [ 6233.017260] Key type ._llcrypt registered [ 6233.019798] Key type .llcrypt registered [ 6233.109489] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6233.904689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6233.925875] alg: No test for adler32 (adler32-zlib) [ 6234.870349] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6235.020711] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6236.647458] Key type lgssc registered [ 6237.439938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6242.876785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6242.898168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6244.228653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.992231] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6254.562172] LustreError: 277846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.575463] LustreError: 277846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6255.263204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.552579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.586749] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6264.104628] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6268.773331] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6274.753967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6275.198183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6280.001846] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6286.033424] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6289.526452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6300.131425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6300.140038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.144049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.180895] Lustre: Skipped 1 previous similar message [ 6305.145562] Lustre: server umount lustre-OST0000 complete [ 6310.368863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.391827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6310.399331] Lustre: Skipped 2 previous similar messages [ 6314.326762] Lustre: server umount lustre-MDT0000 complete [ 6315.488284] LustreError: 277844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.520118] LustreError: 277844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6317.517866] LustreError: 279715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909271 with bad export cookie 11902392758230911403 [ 6317.528748] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6317.865656] Lustre: server umount lustre-MDT0001 complete [ 6326.872660] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6329.168832] Key type lgssc unregistered [ 6329.408161] LNet: 281546:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6329.420543] LNetError: 281546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6329.438369] LNet: Removed LNI 192.168.204.115@tcp [ 6330.218418] Key type .llcrypt unregistered [ 6330.225149] Key type ._llcrypt unregistered [ 6340.776852] Key type ._llcrypt registered [ 6340.779623] Key type .llcrypt registered [ 6340.877833] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6356.363686] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6370.774205] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6372.014136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6372.075022] alg: No test for adler32 (adler32-zlib) [ 6373.126685] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6373.343712] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6374.999249] Key type lgssc registered [ 6375.940599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6404.491619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6404.509396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.720672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6405.753896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6405.826162] Lustre: lustre-MDT0000: new disk, initializing [ 6405.899713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6405.912799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6409.296425] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6417.083416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.192735] Lustre: 285097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6417.232147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6417.236354] Lustre: Skipped 1 previous similar message [ 6417.293798] Lustre: lustre-MDT0001: new disk, initializing [ 6417.371086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6417.410459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6417.422943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6420.787872] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6428.393713] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6432.468764] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6437.130876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6437.346731] Lustre: lustre-OST0000: new disk, initializing [ 6437.350102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6437.355929] Lustre: 287064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6437.421570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.814790] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6446.104386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6446.111258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6446.222600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6449.017805] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6452.451852] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6452.630156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6455.924059] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6456.073493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6461.412871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6461.420779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.440231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6461.922768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6461.936466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.948447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6463.722390] Lustre: server umount lustre-OST0000 complete [ 6467.043578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.055617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6467.059615] Lustre: Skipped 1 previous similar message [ 6472.161479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6472.165740] Lustre: Skipped 1 previous similar message [ 6472.480229] Lustre: server umount lustre-MDT0000 complete [ 6475.124323] LustreError: 285087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909429 with bad export cookie 6638033576352626347 [ 6475.126645] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6475.131588] LustreError: 285087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6475.377072] Lustre: server umount lustre-MDT0001 complete [ 6480.893761] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:17:14 (1788909434) [ 6485.933162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6486.430592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.564200] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6491.619523] LustreError: 289474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.636245] LustreError: 289474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6495.729431] LustreError: 289474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.213386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.481269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6499.920630] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6505.710270] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6509.596731] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6514.741796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6514.979698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6519.649130] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6525.783828] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6541.792195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6541.801232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.802840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6541.823925] Lustre: Skipped 2 previous similar messages [ 6541.833029] Lustre: Skipped 1 previous similar message [ 6544.424121] Lustre: server umount lustre-OST0000 complete [ 6547.937308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6547.947950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.032310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6552.037911] Lustre: Skipped 2 previous similar messages [ 6553.743272] Lustre: server umount lustre-MDT0000 complete [ 6556.470485] LustreError: 291967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909510 with bad export cookie 6638033576352627229 [ 6556.476130] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6556.479584] LustreError: 291967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6556.731278] Lustre: server umount lustre-MDT0001 complete [ 6564.171333] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6566.017041] Key type lgssc unregistered [ 6566.243689] LNet: 293028:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6566.251100] LNetError: 293028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6566.268300] LNet: Removed LNI 192.168.204.115@tcp [ 6566.970925] Key type .llcrypt unregistered [ 6566.973042] Key type ._llcrypt unregistered [ 6586.441869] Key type ._llcrypt registered [ 6586.445543] Key type .llcrypt registered [ 6586.538994] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6596.764243] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6597.527816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6597.615036] alg: No test for adler32 (adler32-zlib) [ 6598.617792] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6598.785807] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6600.447192] Key type lgssc registered [ 6601.254794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6633.454296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6633.471840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.662420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6634.696399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6634.764715] Lustre: lustre-MDT0000: new disk, initializing [ 6634.811046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.830948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6637.752693] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6645.703604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.788995] Lustre: 296831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6645.826091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6645.829823] Lustre: Skipped 1 previous similar message [ 6645.891444] Lustre: lustre-MDT0001: new disk, initializing [ 6645.966119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6645.997992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6646.005359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6649.136699] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6655.805828] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6659.769676] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6664.517435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6664.762732] Lustre: lustre-OST2710: new disk, initializing [ 6664.766279] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6664.771985] Lustre: 298797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6664.846157] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6669.110931] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6670.371353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6670.378699] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6670.450568] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6675.656828] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6679.782531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.892123] Lustre: lustre-OST03e8: new disk, initializing [ 6679.894980] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6679.898953] Lustre: 299861:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6679.953375] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6683.313842] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6685.205131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6685.209130] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6685.277782] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6687.859689] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6694.470707] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6694.634644] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6697.583520] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6697.764432] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6700.697268] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6700.857542] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6703.844894] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6704.006283] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6707.681126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6707.692290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.702242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.756835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.759750] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.282855] Lustre: server umount lustre-MDT0000 complete [ 6716.291631] LustreError: 299860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909670 with bad export cookie 13439415596026319013 [ 6716.292413] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6716.303683] LustreError: 299860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6716.535789] Lustre: server umount lustre-MDT0001 complete [ 6730.283736] Lustre: server umount lustre-OST2710 complete [ 6743.264481] Lustre: server umount lustre-OST03e8 complete [ 6755.643925] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 6761.699989] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6788.496647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.718824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6788.771304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6788.845323] Lustre: lustre-MDT0000: new disk, initializing [ 6788.913989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.926316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6792.593404] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6801.593704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6801.696770] Lustre: 305107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6801.743560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6801.748493] Lustre: Skipped 1 previous similar message [ 6801.803452] Lustre: lustre-MDT0001: new disk, initializing [ 6801.876637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6801.909458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6801.922560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6805.418729] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6813.045815] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6816.800206] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6821.128477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6821.318846] Lustre: lustre-OST0000: new disk, initializing [ 6821.321379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6821.325512] Lustre: 307076:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6821.380931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6825.455734] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6830.629161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6830.642140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6830.753121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6832.693781] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6835.835424] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6835.988630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6838.888364] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6839.081978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6841.318051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.321859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6841.322931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6841.322939] Lustre: Skipped 3 previous similar messages [ 6841.335275] Lustre: Skipped 3 previous similar messages [ 6846.491938] Lustre: server umount lustre-OST0000 complete [ 6851.553529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.555314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6851.565262] Lustre: Skipped 1 previous similar message [ 6851.567871] Lustre: Skipped 1 previous similar message [ 6856.673071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6856.682679] Lustre: Skipped 1 previous similar message [ 6863.328070] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6863.494864] Lustre: server umount lustre-MDT0000 complete [ 6865.888233] LustreError: 305120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.902944] LustreError: 305120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6866.157255] LustreError: 305099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909820 with bad export cookie 13439415596026320693 [ 6866.162743] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6866.170254] LustreError: 305099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6866.374162] Lustre: server umount lustre-MDT0001 complete [ 6872.023412] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:23:45 (1788909825) [ 6873.175549] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6874.400692] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:23:48 (1788909828) [ 6876.542653] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6894.589421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.745251] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6894.759637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6894.859800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6897.474213] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6903.097614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6903.176796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6903.199896] Lustre: 310935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6906.058254] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 6924.581612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6924.678770] Lustre: Denying initial registration attempt from nid 192.168.204.115@tcp, specified as failover [ 6924.684611] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6924.690555] LustreError: 312389:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -99 [ 6924.694891] LustreError: 312389:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 6924.699395] LustreError: 312389:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6924.742665] Lustre: server umount lustre-OST0000 complete [ 6924.745504] LustreError: 312389:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6931.937103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.938571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6931.945180] Lustre: Skipped 1 previous similar message [ 6931.948237] Lustre: Skipped 2 previous similar messages [ 6935.381492] Lustre: server umount lustre-MDT0000 complete [ 6937.057229] LustreError: 310947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6937.761440] LustreError: 310927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909892 with bad export cookie 13439415596026321575 [ 6937.766718] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6937.769157] LustreError: 310927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6937.960462] Lustre: server umount lustre-MDT0001 complete [ 6944.469306] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 6946.283471] Key type lgssc unregistered [ 6946.478260] LNet: 313508:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6946.481824] LNetError: 313508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6946.490734] LNet: Removed LNI 192.168.204.115@tcp [ 6947.006845] Key type .llcrypt unregistered [ 6947.011531] Key type ._llcrypt unregistered [ 6953.077742] Key type ._llcrypt registered [ 6953.079267] Key type .llcrypt registered [ 6953.131080] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6953.681247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6953.701095] alg: No test for adler32 (adler32-zlib) [ 6954.657274] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6954.821417] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 6956.431676] Key type lgssc registered [ 6957.092267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6990.317145] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 6995.514989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6995.530985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.680849] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6996.698427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6996.800700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6999.045990] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7004.208555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.262281] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7004.280763] Lustre: 316249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7004.433720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7006.846441] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7011.493661] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7014.988341] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7032.166399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7032.285082] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7032.400888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.802792] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7040.202458] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7043.042876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7043.044515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.048334] LustreError: Skipped 1 previous similar message [ 7043.048995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7043.063819] Lustre: Skipped 1 previous similar message [ 7047.879937] Lustre: server umount lustre-OST0000 complete [ 7050.721937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7050.728871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.738797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.743452] Lustre: Skipped 1 previous similar message [ 7053.281705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7053.282027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7056.461676] Lustre: server umount lustre-MDT0000 complete [ 7058.679874] LustreError: 319068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910013 with bad export cookie 15425670240410667199 [ 7058.682471] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7058.687248] LustreError: 319068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7058.866115] Lustre: server umount lustre-MDT0001 complete [ 7064.908487] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7066.522295] Key type lgssc unregistered [ 7066.713226] LNet: 319818:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7066.719339] LNetError: 319818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7066.734801] LNet: Removed LNI 192.168.204.115@tcp [ 7067.223786] Key type .llcrypt unregistered [ 7067.225786] Key type ._llcrypt unregistered [ 7079.059805] Key type ._llcrypt registered [ 7079.062388] Key type .llcrypt registered [ 7079.122979] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7079.307321] Key type .llcrypt unregistered [ 7079.310338] Key type ._llcrypt unregistered [ 7088.596142] Key type ._llcrypt registered [ 7088.598202] Key type .llcrypt registered [ 7088.664949] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7089.345482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7089.361343] alg: No test for adler32 (adler32-zlib) [ 7090.279777] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7090.419037] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7092.023245] Key type lgssc registered [ 7092.715938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7119.069303] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 7119.967357] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 7127.900208] loop0: detected capacity change from 0 to 204800000 [ 7128.090846] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7128.098094] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7128.111378] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7143.582497] loop0: detected capacity change from 0 to 204800000 [ 7143.709452] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7143.716992] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7143.723972] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7159.315561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7159.331507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7160.487205] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7160.499214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7160.599435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7163.228934] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7168.318316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7168.378967] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7168.400745] Lustre: 323719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7168.552717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7171.371709] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7175.352623] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7178.693584] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7181.916105] loop0: detected capacity change from 0 to 204800000 [ 7198.695034] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.857834] Lustre: sv13738-OST0000: new disk, initializing [ 7198.861664] Lustre: srv-sv13738-OST0000: No data found on store. Initialize space. [ 7198.866948] Lustre: 325797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7198.911574] Lustre: sv13738-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.601322] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7209.581753] loop1: detected capacity change from 0 to 204800000 [ 7226.604362] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7226.705172] Lustre: sv13738-OST0001: new disk, initializing [ 7226.710290] Lustre: srv-sv13738-OST0001: No data found on store. Initialize space. [ 7226.715485] Lustre: 326914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7226.757173] Lustre: sv13738-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7231.255516] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7240.247934] Lustre: server umount sv13738-OST0001 complete [ 7246.593289] Lustre: server umount sv13738-OST0000 complete [ 7255.522416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7255.528133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.549556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7256.034065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.720303] Lustre: server umount lustre-MDT0000 complete [ 7263.372958] LustreError: 324461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910217 with bad export cookie 15078168506222206499 [ 7263.376973] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.382257] LustreError: 324461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7263.621842] Lustre: server umount lustre-MDT0001 complete [ 7271.626238] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7273.381144] Key type lgssc unregistered [ 7273.589683] LNet: 329665:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7273.598608] LNetError: 329665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7273.606772] LNet: Removed LNI 192.168.204.115@tcp [ 7274.664533] LNet: Removed LNI 192.168.204.115@tcp1 [ 7275.266771] Key type .llcrypt unregistered [ 7275.269168] Key type ._llcrypt unregistered [ 7286.848112] Key type ._llcrypt registered [ 7286.849923] Key type .llcrypt registered [ 7286.911862] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7287.094432] Key type .llcrypt unregistered [ 7287.096542] Key type ._llcrypt unregistered [ 7297.165120] Key type ._llcrypt registered [ 7297.169076] Key type .llcrypt registered [ 7297.232803] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7297.820098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7297.861054] alg: No test for adler32 (adler32-zlib) [ 7298.780934] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7298.928868] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7300.543185] Key type lgssc registered [ 7301.214783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7314.463563] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 7315.256693] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 7321.364913] loop0: detected capacity change from 0 to 204800000 [ 7321.480651] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7321.491478] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7321.503155] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7337.034600] loop0: detected capacity change from 0 to 204800000 [ 7337.157460] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7337.164850] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7337.175289] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7351.977770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7351.991924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.163453] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7353.182656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7353.287334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7355.591387] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7360.645850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.731223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7360.757655] Lustre: 333463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7360.937566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7363.366494] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7367.613923] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7370.583097] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7373.111605] loop0: detected capacity change from 0 to 204800000 [ 7389.285993] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.471873] Lustre: sv32036-OST0000: new disk, initializing [ 7389.475446] Lustre: srv-sv32036-OST0000: No data found on store. Initialize space. [ 7389.479605] Lustre: 335538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7389.518266] Lustre: sv32036-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7392.995475] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7399.087360] loop1: detected capacity change from 0 to 204800000 [ 7415.588693] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7415.707737] Lustre: sv32036-OST0001: new disk, initializing [ 7415.710971] Lustre: srv-sv32036-OST0001: No data found on store. Initialize space. [ 7415.715350] Lustre: 336656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7415.756195] Lustre: sv32036-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7418.841161] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7425.691565] Lustre: server umount sv32036-OST0001 complete [ 7430.574254] Lustre: server umount sv32036-OST0000 complete [ 7440.353690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7440.359751] Lustre: Skipped 1 previous similar message [ 7440.362384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7440.365845] Lustre: Skipped 1 previous similar message [ 7444.377728] Lustre: server umount lustre-MDT0000 complete [ 7446.652535] LustreError: 333454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910401 with bad export cookie 4142259454138067733 [ 7446.655514] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7446.666116] LustreError: 333454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7446.831645] Lustre: server umount lustre-MDT0001 complete [ 7453.454386] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7454.904872] Key type lgssc unregistered [ 7455.037140] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.15@tcp [ 7455.057827] LNet: 339406:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7455.063195] LNetError: 339406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7455.068378] LNet: Removed LNI 192.168.204.115@tcp [ 7456.103304] LNet: Removed LNI 192.168.204.115@tcp1 [ 7456.549137] Key type .llcrypt unregistered [ 7456.550823] Key type ._llcrypt unregistered [ 7465.054957] Key type ._llcrypt registered [ 7465.057756] Key type .llcrypt registered [ 7465.106183] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7465.277438] Key type .llcrypt unregistered [ 7465.279148] Key type ._llcrypt unregistered [ 7474.108597] Key type ._llcrypt registered [ 7474.110135] Key type .llcrypt registered [ 7474.173814] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7474.601863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7474.613101] alg: No test for adler32 (adler32-zlib) [ 7475.516763] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7475.639170] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7477.247337] Key type lgssc registered [ 7477.795460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7487.947911] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 7488.509197] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 7493.360510] loop0: detected capacity change from 0 to 204800000 [ 7493.529633] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7493.536194] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7493.543589] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7507.535031] loop0: detected capacity change from 0 to 204800000 [ 7507.664012] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7507.671377] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7507.677976] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7522.435205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7522.446794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.580698] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7523.592703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7523.672642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.822309] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7529.964260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.014494] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7530.035595] Lustre: 343204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7530.174219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7532.233974] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7535.907772] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7538.412618] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7540.816415] loop0: detected capacity change from 0 to 204800000 [ 7556.321562] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7556.432761] Lustre: sv28984-OST0000: new disk, initializing [ 7556.434916] Lustre: srv-sv28984-OST0000: No data found on store. Initialize space. [ 7556.437945] Lustre: 345276:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7556.465427] Lustre: sv28984-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.237483] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7567.950478] Lustre: server umount sv28984-OST0000 complete [ 7576.546910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7576.552345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7576.559439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7578.593957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7578.594057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.837457] Lustre: server umount lustre-MDT0000 complete [ 7582.996193] LustreError: 343193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910537 with bad export cookie 15200768388451160088 [ 7583.002647] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7583.005593] LustreError: 343193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7583.201354] Lustre: server umount lustre-MDT0001 complete [ 7589.833863] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7591.373151] Key type lgssc unregistered [ 7591.566775] LNet: 347727:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7591.573698] LNetError: 347727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7591.575902] LNet: Removed LNI 192.168.204.115@tcp [ 7592.615681] LNet: Removed LNI 192.168.204.115@tcp1 [ 7593.166038] Key type .llcrypt unregistered [ 7593.167590] Key type ._llcrypt unregistered [ 7603.412786] Key type ._llcrypt registered [ 7603.414760] Key type .llcrypt registered [ 7603.483181] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7603.645232] Key type .llcrypt unregistered [ 7603.647521] Key type ._llcrypt unregistered [ 7613.471617] Key type ._llcrypt registered [ 7613.473630] Key type .llcrypt registered [ 7613.540622] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7614.176470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7614.194241] alg: No test for adler32 (adler32-zlib) [ 7615.142662] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7615.289307] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7616.903189] Key type lgssc registered [ 7617.622053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7630.466545] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 7631.177403] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 7636.582155] loop0: detected capacity change from 0 to 204800000 [ 7636.651052] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7636.660185] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7636.665986] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7651.230096] loop0: detected capacity change from 0 to 204800000 [ 7651.341164] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7651.349045] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7651.356961] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7666.303869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7666.320425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7667.489600] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7667.506822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7667.598658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7670.207485] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7674.914163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7674.976430] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7674.998941] Lustre: 351571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7675.184326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7677.212645] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7680.593637] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7683.201210] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7685.779232] loop0: detected capacity change from 0 to 204800000 [ 7701.857996] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7701.983847] Lustre: sv2180-OST0000: new disk, initializing [ 7701.987152] Lustre: srv-sv2180-OST0000: No data found on store. Initialize space. [ 7701.991384] Lustre: 353646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7702.021673] Lustre: sv2180-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7704.704608] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7712.187840] Lustre: server umount sv2180-OST0000 complete [ 7721.440731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.440850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7721.442156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7721.448902] Lustre: Skipped 1 previous similar message [ 7726.560389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7726.565433] Lustre: Skipped 2 previous similar messages [ 7731.681176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.684527] Lustre: Skipped 1 previous similar message [ 7732.191113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7732.290189] Lustre: server umount lustre-MDT0000 complete [ 7734.390265] LustreError: 352312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910688 with bad export cookie 4275562480812716107 [ 7734.393717] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7734.397502] LustreError: 352312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7734.567581] Lustre: server umount lustre-MDT0001 complete [ 7740.825617] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7742.295975] Key type lgssc unregistered [ 7742.458207] LNet: 356101:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7742.463088] LNetError: 356101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7742.467157] LNet: Removed LNI 192.168.204.115@tcp [ 7743.525691] LNet: Removed LNI 192.168.204.115@tcp1 [ 7744.059336] Key type .llcrypt unregistered [ 7744.061128] Key type ._llcrypt unregistered [ 7753.127299] Key type ._llcrypt registered [ 7753.128654] Key type .llcrypt registered [ 7753.180246] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7753.320616] Key type .llcrypt unregistered [ 7753.321891] Key type ._llcrypt unregistered [ 7762.425356] Key type ._llcrypt registered [ 7762.427083] Key type .llcrypt registered [ 7762.535488] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7763.099973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7763.130187] alg: No test for adler32 (adler32-zlib) [ 7764.105631] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7764.303478] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7765.951282] Key type lgssc registered [ 7766.904522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7787.559572] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 7788.793192] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 7798.410695] loop0: detected capacity change from 0 to 204800000 [ 7798.601234] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7798.618102] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7798.649067] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7813.417319] loop0: detected capacity change from 0 to 204800000 [ 7813.607516] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7813.618104] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7813.635430] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7828.665796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7828.687841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.854779] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7829.873701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7829.970658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7832.361868] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7838.174979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.332854] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7838.364791] Lustre: 359899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7838.537019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7842.218558] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7848.938866] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7854.264569] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7858.098525] loop0: detected capacity change from 0 to 204800000 [ 7874.921313] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7875.056426] Lustre: sv3627-OST0000: new disk, initializing [ 7875.059254] Lustre: srv-sv3627-OST0000: No data found on store. Initialize space. [ 7875.063319] Lustre: 361977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7875.097457] Lustre: sv3627-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.234304] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7884.207282] loop1: detected capacity change from 0 to 204800000 [ 7900.264952] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7900.367504] Lustre: sv3627-OST0001: new disk, initializing [ 7900.373923] Lustre: srv-sv3627-OST0001: No data found on store. Initialize space. [ 7900.378289] Lustre: 363145:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7900.414843] Lustre: sv3627-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7903.686071] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 7910.779200] Lustre: server umount sv3627-OST0001 complete [ 7915.727979] Lustre: server umount sv3627-OST0000 complete [ 7922.656737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.658427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7922.669185] Lustre: Skipped 1 previous similar message [ 7922.681990] Lustre: Skipped 1 previous similar message [ 7927.777712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7927.781872] Lustre: Skipped 1 previous similar message [ 7928.505646] Lustre: server umount lustre-MDT0000 complete [ 7930.533217] LustreError: 360642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910884 with bad export cookie 7694688192152942891 [ 7930.535249] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7930.538148] LustreError: 360642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7930.727486] Lustre: server umount lustre-MDT0001 complete [ 7937.390771] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7939.023348] Key type lgssc unregistered [ 7939.172111] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.15@tcp [ 7939.201467] LNet: 365897:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7939.206896] LNetError: 365897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7939.215429] LNet: Removed LNI 192.168.204.115@tcp [ 7940.269476] LNet: Removed LNI 192.168.204.115@tcp1 [ 7940.842892] Key type .llcrypt unregistered [ 7940.845427] Key type ._llcrypt unregistered [ 7950.395195] Key type ._llcrypt registered [ 7950.397234] Key type .llcrypt registered [ 7950.462860] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7950.652534] Key type .llcrypt unregistered [ 7950.655301] Key type ._llcrypt unregistered [ 7958.908348] Key type ._llcrypt registered [ 7958.909733] Key type .llcrypt registered [ 7958.968944] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 7959.474884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7959.495611] alg: No test for adler32 (adler32-zlib) [ 7960.395988] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7960.514909] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 7962.127374] Key type lgssc registered [ 7962.648062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7973.112069] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 7973.757455] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 7987.737402] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7989.162087] Key type lgssc unregistered [ 7989.312772] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.15@tcp [ 7989.334525] LNet: 369829:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7989.338982] LNetError: 369829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7989.345872] LNet: Removed LNI 192.168.204.115@tcp [ 7989.352785] LNet: Removed LNI 192.168.204.115@tcp1 [ 7989.767766] Key type .llcrypt unregistered [ 7989.769828] Key type ._llcrypt unregistered [ 7998.313458] Key type ._llcrypt registered [ 7998.314695] Key type .llcrypt registered [ 7998.372920] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 7998.508273] Key type .llcrypt unregistered [ 7998.510770] Key type ._llcrypt unregistered [ 8007.141904] Key type ._llcrypt registered [ 8007.144425] Key type .llcrypt registered [ 8007.195155] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8007.772372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8007.784312] alg: No test for adler32 (adler32-zlib) [ 8008.699249] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8008.830313] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8010.455718] Key type lgssc registered [ 8011.019851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8021.517530] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 8022.172550] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 8023.045450] LNet: Added LNI 192.168.204.115@tcp2 [8/256/0/180] [ 8028.199133] loop0: detected capacity change from 0 to 204800000 [ 8028.345515] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8028.354131] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8028.360621] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8042.864919] loop0: detected capacity change from 0 to 204800000 [ 8042.988801] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8042.997965] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8043.008374] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8056.872173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8056.884644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8057.949875] LNetError: 371132:0:(peer.c:2149:lnet_peer_add_nid()) Peer 192.168.204.115@tcp NID 192.168.204.115@tcp is already locked with peer 192.168.204.115@tcp [ 8058.015456] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.115@tcp [ 8058.017389] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8058.033380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8058.124658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8059.946877] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8064.041359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8064.110656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8064.127972] Lustre: 373713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8064.235988] LNetError: 371132:0:(peer.c:2149:lnet_peer_add_nid()) Peer 192.168.204.115@tcp NID 192.168.204.115@tcp is already locked with peer 192.168.204.115@tcp [ 8064.247494] LNetError: 371132:0:(peer.c:2149:lnet_peer_add_nid()) Skipped 1 previous similar message [ 8064.287085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8066.223913] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8069.751306] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8072.242239] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8074.977320] loop0: detected capacity change from 0 to 204800000 [ 8090.980221] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.107894] Lustre: sv28284-OST0000: new disk, initializing [ 8091.111085] Lustre: srv-sv28284-OST0000: No data found on store. Initialize space. [ 8091.117698] Lustre: 375791:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8091.153402] Lustre: sv28284-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.247845] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8099.491426] loop1: detected capacity change from 0 to 204800000 [ 8114.981331] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8115.048575] Lustre: sv28284-OST0001: new disk, initializing [ 8115.051719] Lustre: srv-sv28284-OST0001: No data found on store. Initialize space. [ 8115.056543] Lustre: 376960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8115.086201] Lustre: sv28284-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8117.509388] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8123.503625] Lustre: server umount sv28284-OST0001 complete [ 8127.702934] Lustre: server umount sv28284-OST0000 complete [ 8136.160555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8136.164092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.171224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8136.672396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.678797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8139.233697] Lustre: server umount lustre-MDT0000 complete [ 8140.939730] LustreError: 374456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911095 with bad export cookie 10447069874980961692 [ 8140.943059] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8140.944954] LustreError: 374456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8141.073814] Lustre: server umount lustre-MDT0001 complete [ 8146.645618] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8147.883056] Key type lgssc unregistered [ 8148.045454] LNet: 379709:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8148.048974] LNetError: 379709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8148.052606] LNet: Removed LNI 192.168.204.115@tcp [ 8149.087863] LNet: Removed LNI 192.168.204.115@tcp1 [ 8149.090171] LNetError: 379709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8149.094024] LNetError: 379709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8149.103574] LNet: Removed LNI 192.168.204.115@tcp2 [ 8149.475348] Key type .llcrypt unregistered [ 8149.477177] Key type ._llcrypt unregistered [ 8157.629474] Key type ._llcrypt registered [ 8157.630760] Key type .llcrypt registered [ 8157.681238] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8157.810373] Key type .llcrypt unregistered [ 8157.812671] Key type ._llcrypt unregistered [ 8165.306288] Key type ._llcrypt registered [ 8165.308050] Key type .llcrypt registered [ 8165.363472] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8165.763267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8165.778523] alg: No test for adler32 (adler32-zlib) [ 8166.654588] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8166.765083] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8168.359223] Key type lgssc registered [ 8168.837800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8177.574374] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 8178.109304] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 8182.453236] loop0: detected capacity change from 0 to 204800000 [ 8182.525161] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8182.529538] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8182.535194] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8195.982915] loop0: detected capacity change from 0 to 204800000 [ 8196.055452] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.060584] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.065700] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8210.116391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8210.126511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.233681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8211.241871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8211.313646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8213.035520] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8216.422955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.458598] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8216.470601] Lustre: 383555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8216.573934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8218.033865] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8220.870446] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8223.004897] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8225.202930] loop0: detected capacity change from 0 to 204800000 [ 8240.867119] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8240.979758] Lustre: sv8336-OST0000: new disk, initializing [ 8240.982750] Lustre: srv-sv8336-OST0000: No data found on store. Initialize space. [ 8240.987090] Lustre: 385630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8241.028149] Lustre: sv8336-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.699448] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8248.252675] loop1: detected capacity change from 0 to 204800000 [ 8263.458759] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.521321] Lustre: sv8336-OST0001: new disk, initializing [ 8263.523975] Lustre: srv-sv8336-OST0001: No data found on store. Initialize space. [ 8263.528291] Lustre: 386746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8263.557644] Lustre: sv8336-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8266.158126] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8272.198375] Lustre: server umount sv8336-OST0001 complete [ 8276.300188] Lustre: server umount sv8336-OST0000 complete [ 8283.103991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8283.107416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.113058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8283.617215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.624522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8288.736326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8288.739245] Lustre: Skipped 1 previous similar message [ 8293.857418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.860433] Lustre: Skipped 1 previous similar message [ 8295.391129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8295.462881] Lustre: server umount lustre-MDT0000 complete [ 8296.950409] LustreError: 383590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911251 with bad export cookie 14296524638120270201 [ 8296.953562] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8296.956428] LustreError: 383590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8297.080764] Lustre: server umount lustre-MDT0001 complete [ 8302.392925] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8303.547829] Key type lgssc unregistered [ 8303.669284] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.15@tcp [ 8303.694592] LNet: 389495:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8303.698439] LNetError: 389495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8303.701654] LNet: Removed LNI 192.168.204.115@tcp [ 8304.741399] LNet: Removed LNI 192.168.204.115@tcp1 [ 8305.102404] Key type .llcrypt unregistered [ 8305.105842] Key type ._llcrypt unregistered [ 8311.985283] Key type ._llcrypt registered [ 8311.987722] Key type .llcrypt registered [ 8312.036699] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8312.163602] Key type .llcrypt unregistered [ 8312.166285] Key type ._llcrypt unregistered [ 8314.525341] Key type ._llcrypt registered [ 8314.526818] Key type .llcrypt registered [ 8314.565435] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8314.675502] Key type .llcrypt unregistered [ 8314.677335] Key type ._llcrypt unregistered [ 8319.333147] Key type ._llcrypt registered [ 8319.334307] Key type .llcrypt registered [ 8319.374129] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8319.493378] Key type .llcrypt unregistered [ 8319.494898] Key type ._llcrypt unregistered [ 8325.512501] Key type ._llcrypt registered [ 8325.513876] Key type .llcrypt registered [ 8325.553722] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8325.665737] Key type .llcrypt unregistered [ 8325.666882] Key type ._llcrypt unregistered [ 8332.739703] Key type ._llcrypt registered [ 8332.744645] Key type .llcrypt registered [ 8332.794781] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8333.295381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8333.335511] alg: No test for adler32 (adler32-zlib) [ 8334.212673] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8334.317548] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8335.919337] Key type lgssc registered [ 8336.367690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8343.850433] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [ 8344.277337] LNet: Added LNI 192.168.204.115@tcp1 [8/256/0/180] [ 8347.781708] loop0: detected capacity change from 0 to 204800000 [ 8347.844190] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8347.848367] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8347.853902] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8361.220929] loop0: detected capacity change from 0 to 204800000 [ 8361.296821] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8361.302745] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8361.307172] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8374.874585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8374.881641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.984072] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8375.997893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8376.074180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.762109] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8380.967747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8381.014363] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8381.030568] Lustre: 394919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8381.144263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.819555] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8385.687944] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8387.873712] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8389.837867] loop0: detected capacity change from 0 to 204800000 [ 8405.025759] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8405.125235] Lustre: sv9338-OST0000: new disk, initializing [ 8405.127480] Lustre: srv-sv9338-OST0000: No data found on store. Initialize space. [ 8405.130454] Lustre: 396994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8405.151504] Lustre: sv9338-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.371980] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8410.769451] Lustre: server umount sv9338-OST0000 complete [ 8415.353496] loop0: detected capacity change from 0 to 204800000 [ 8419.298324] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8419.404699] Lustre: *** cfs_fail_loc=727, val=1*** [ 8419.410039] Lustre: sv9338-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.555820] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8427.201676] Lustre: server umount sv9338-OST0000 complete [ 8432.608070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8432.610837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.617219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8435.168195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8435.170276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.761297] Lustre: server umount lustre-MDT0000 complete [ 8439.411514] LustreError: 400405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911393 with bad export cookie 4128094721494206553 [ 8439.413774] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8439.418053] LustreError: 400405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8439.560373] Lustre: server umount lustre-MDT0001 complete [ 8444.362460] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8445.807163] Key type lgssc unregistered [ 8445.967611] LNet: 401155:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8445.971228] LNetError: 401155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8445.974883] LNet: Removed LNI 192.168.204.115@tcp [ 8447.018165] LNet: Removed LNI 192.168.204.115@tcp1 [ 8447.386248] Key type .llcrypt unregistered [ 8447.387537] Key type ._llcrypt unregistered [ 8456.205552] Key type ._llcrypt registered [ 8456.206502] Key type .llcrypt registered [ 8456.249635] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8456.586079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8456.593063] alg: No test for adler32 (adler32-zlib) [ 8457.466334] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8457.569331] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8459.167221] Key type lgssc registered [ 8459.630400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8462.725494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8462.743761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8463.923566] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8463.941586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8465.724556] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8469.000867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.105962] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8469.110634] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 8469.151815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8470.814547] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8473.647594] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8475.822823] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8480.355425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8480.427851] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8480.508358] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8480.512090] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 8480.526090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8482.607205] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8485.303643] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8489.450908] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8489.455348] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 8494.562369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8494.566365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8494.568352] Lustre: Skipped 1 previous similar message [ 8496.636488] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 8496.639802] Lustre: Skipped 2 previous similar messages [ 8499.680920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.683692] Lustre: Skipped 2 previous similar messages [ 8499.741911] Lustre: server umount lustre-MDT0000 complete [ 8500.985097] LustreError: 402433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911455 with bad export cookie 6971290175667766887 [ 8500.987941] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8500.989024] LustreError: 402433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8501.745512] LustreError: 404944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8501.753912] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 8501.754418] LustreError: 404944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8504.800300] LustreError: 403163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.800664] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.811049] Lustre: Skipped 1 previous similar message [ 8505.823885] LustreError: 402450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.824220] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8505.831451] Lustre: Skipped 1 previous similar message [ 8507.185948] Lustre: server umount lustre-MDT0001 complete [ 8509.492784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8512.508773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.303632] LustreError: 401762:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d7376ae6 x1875809571727872/t0(0) o250->MGC192.168.204.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8526.418715] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8526.421915] LNetError: 401761:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [ 8526.427334] LustreError: 404279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.434130] LustreError: 404279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8526.453538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.483338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8528.033346] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8531.378199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.442102] LustreError: 406055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.554613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8532.581089] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8533.082662] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8535.653519] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8535.771567] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8537.332219] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 8539.248245] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8539.820080] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8541.664605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8541.667595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.672345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8541.674814] Lustre: Skipped 1 previous similar message [ 8542.176066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8547.286603] Lustre: server umount lustre-OST0000 complete [ 8551.393344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.400034] Lustre: Skipped 2 previous similar messages [ 8554.648912] Lustre: server umount lustre-MDT0000 complete [ 8556.021198] LustreError: 402477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911510 with bad export cookie 6971290175667774132 [ 8556.024113] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.026605] LustreError: 402477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8556.152213] Lustre: server umount lustre-MDT0001 complete [ 8560.153867] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8561.305837] Key type lgssc unregistered [ 8561.448349] LNet: 408723:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8561.452697] LNetError: 408723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8561.461481] LNet: Removed LNI 192.168.204.115@tcp [ 8561.792057] Key type .llcrypt unregistered [ 8561.793543] Key type ._llcrypt unregistered [ 8571.714439] Key type ._llcrypt registered [ 8571.715475] Key type .llcrypt registered [ 8571.765954] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8572.204441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8572.252052] alg: No test for adler32 (adler32-zlib) [ 8573.148240] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8573.294550] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8574.903925] Key type lgssc registered [ 8575.501336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8578.925025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8578.935651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8580.120217] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8580.140804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8581.594220] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8585.148749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.283627] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8585.288648] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 4 previous similar messages [ 8585.343106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8587.111267] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8589.860791] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8592.110364] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8595.424771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.425659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8595.426119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.429503] Lustre: Skipped 1 previous similar message [ 8599.511735] Lustre: server umount lustre-MDT0000 complete [ 8600.543915] LustreError: 410844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.550803] LustreError: 410844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8600.843349] LustreError: 410098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911555 with bad export cookie 1242418539132461154 [ 8600.846109] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8600.848464] LustreError: 410098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8600.981967] Lustre: server umount lustre-MDT0001 complete [ 8606.569251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8606.632947] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8606.636929] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 4 previous similar messages [ 8606.697644] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8606.712114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8606.808449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.443716] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8611.935632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.987114] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8612.001153] Lustre: 412700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8612.092726] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8612.097483] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 8612.152138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8613.881087] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8616.856774] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8618.475898] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8618.479074] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 8618.859063] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8623.651115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8623.735239] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8623.740625] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8623.745523] LustreError: 414584:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 8623.749392] LustreError: 414584:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 8623.751859] LustreError: 414584:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8623.782551] Lustre: server umount lustre-OST0000 complete [ 8623.784295] LustreError: 414584:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8628.770584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.859279] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8628.927928] LNetError: 409426:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.115@tcp1 has no available nets [ 8628.943923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8630.993249] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8633.648489] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8636.359724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8636.396909] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8636.427918] Lustre: lustre-OST0001: new disk, initializing [ 8636.430118] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8636.433538] Lustre: 416113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8638.444728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8638.451322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8638.484259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8638.520427] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8638.744759] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8641.538643] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8643.557199] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.558379] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8643.561933] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8643.563904] LustreError: Skipped 1 previous similar message [ 8643.568292] Lustre: Skipped 2 previous similar messages [ 8644.615497] Lustre: server umount lustre-OST0001 complete [ 8648.674061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8648.676362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.676801] LustreError: 415052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.679158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8648.681458] Lustre: Skipped 2 previous similar messages [ 8652.253944] Lustre: server umount lustre-OST0000 complete [ 8653.792117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.798634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8653.801097] Lustre: Skipped 1 previous similar message [ 8658.400658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8658.404043] LustreError: Skipped 1 previous similar message [ 8658.406049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8658.414039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8659.832782] Lustre: server umount lustre-MDT0000 complete [ 8661.357656] LustreError: 413442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911615 with bad export cookie 1242418539132461693 [ 8661.362155] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8661.363300] LustreError: 413442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8661.529600] Lustre: server umount lustre-MDT0001 complete [ 8665.448837] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8666.618252] Key type lgssc unregistered [ 8666.752477] LNet: 418071:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8666.756312] LNetError: 418071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8666.766451] LNet: Removed LNI 192.168.204.115@tcp [ 8667.121434] Key type .llcrypt unregistered [ 8667.123244] Key type ._llcrypt unregistered [ 8699.092137] Key type ._llcrypt registered [ 8699.093715] Key type .llcrypt registered [ 8699.139839] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 8705.313082] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8705.724215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8705.743407] alg: No test for adler32 (adler32-zlib) [ 8706.604802] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8706.703297] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8708.287145] Key type lgssc registered [ 8708.669882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8720.788188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8720.794038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.897784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8721.909988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8721.948679] Lustre: lustre-MDT0000: new disk, initializing [ 8721.974103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8721.981125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8723.367648] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8727.865085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8727.906349] Lustre: 422973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8727.933481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8727.935356] Lustre: Skipped 1 previous similar message [ 8727.969544] Lustre: lustre-MDT0001: new disk, initializing [ 8728.009712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8728.023252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8728.027419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8729.377838] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8733.327923] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8735.182357] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8737.568142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8737.688371] Lustre: lustre-OST0000: new disk, initializing [ 8737.691221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8737.694556] Lustre: 424943:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8737.723879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8739.708018] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8743.571733] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8745.108740] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8745.463677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8745.467413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8745.492088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8746.207358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 8747.753785] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8747.830817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8750.561646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8750.562610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8750.563594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8750.564639] LustreError: Skipped 1 previous similar message [ 8750.569637] Lustre: Skipped 1 previous similar message [ 8754.587093] Lustre: server umount lustre-OST0000 complete [ 8758.751818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8758.754339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.758504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8758.760583] Lustre: Skipped 1 previous similar message [ 8760.801290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.801387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8760.808626] Lustre: Skipped 1 previous similar message [ 8762.080251] Lustre: server umount lustre-MDT0000 complete [ 8763.440143] LustreError: 423768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911717 with bad export cookie 3731410730622122072 [ 8763.443735] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8763.445807] LustreError: 423768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8763.574282] Lustre: server umount lustre-MDT0001 complete [ 8766.601546] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:55:20 (1788911720) [ 8769.258396] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:55:23 (1788911723) [ 8772.324740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.524223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.957236] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8776.919933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.074157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8778.596864] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8781.089547] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8783.160371] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8785.859308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8786.019961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8788.361074] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8790.959306] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8804.639611] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8805.173528] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8805.715341] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8806.294854] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8806.949872] Lustre: Failing over lustre-MDT0000 [ 8807.135730] Lustre: server umount lustre-MDT0000 complete [ 8807.908075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.908988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8807.909283] LustreError: 427728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.909294] LustreError: 427728:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8807.914363] Lustre: Skipped 1 previous similar message [ 8811.514190] LustreError: 429542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8811.519156] LustreError: 429542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8813.024577] LustreError: 427704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8813.029912] LustreError: 427704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8816.626064] LustreError: 427700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.246386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.296836] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8820.395085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.419885] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8821.745793] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8821.803227] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8825.835100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8825.855681] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8825.875639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8828.058858] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8828.662895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8830.659307] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8831.382961] Lustre: Failing over lustre-MDT0000 [ 8831.529075] Lustre: server umount lustre-MDT0000 complete [ 8831.995297] LustreError: 427700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8832.005050] LustreError: 427700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8836.066156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8836.066351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.073575] Lustre: Skipped 3 previous similar messages [ 8841.185643] LustreError: 427703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.192837] LustreError: 427703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8845.059148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8845.130728] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8845.274280] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8846.874574] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8847.347571] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8850.408509] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8850.410687] Lustre: Skipped 2 previous similar messages [ 8850.419760] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8850.437036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8852.388111] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8852.947501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8855.523110] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.524688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8855.527872] Lustre: Skipped 2 previous similar messages [ 8855.528295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8855.530056] Lustre: Skipped 2 previous similar messages [ 8860.643378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.649292] Lustre: Skipped 2 previous similar messages [ 8861.488188] Lustre: server umount lustre-MDT0000 complete [ 8863.042679] LustreError: 431685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911817 with bad export cookie 3731410730622124753 [ 8863.046629] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8863.048181] LustreError: 431685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8863.193281] Lustre: server umount lustre-MDT0001 complete [ 8875.058270] Lustre: server umount lustre-OST0000 complete [ 8881.331521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8881.517596] LustreError: 434051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8881.524427] LustreError: 434051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8881.543905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8881.546967] Lustre: Skipped 1 previous similar message [ 8883.224853] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8886.868343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.870233] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8892.008623] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8894.080720] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8896.653608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8898.974638] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8901.768928] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8906.031684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8911.330263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.330485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8911.335986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8911.339668] Lustre: Skipped 1 previous similar message [ 8914.512486] Lustre: server umount lustre-OST0000 complete [ 8930.271335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8930.336786] Lustre: server umount lustre-MDT0000 complete [ 8930.784133] LustreError: 434051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8930.789030] LustreError: 434051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8931.688979] LustreError: 434033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911886 with bad export cookie 3731410730622125474 [ 8931.693323] LustreError: 434033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8931.693601] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8931.823911] Lustre: server umount lustre-MDT0001 complete [ 8935.612200] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 8936.722614] Key type lgssc unregistered [ 8936.867453] LNet: 437582:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8936.870526] LNetError: 437582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8936.882353] LNet: Removed LNI 192.168.204.115@tcp [ 8937.218776] Key type .llcrypt unregistered [ 8937.220022] Key type ._llcrypt unregistered [ 8947.756659] Key type ._llcrypt registered [ 8947.757995] Key type .llcrypt registered [ 8947.798675] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 8948.283273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8948.312625] alg: No test for adler32 (adler32-zlib) [ 8949.171918] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8949.265064] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 8950.839191] Key type lgssc registered [ 8951.242042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8953.977575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8953.984576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.138751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8956.572325] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8959.575188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.706684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8961.135428] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8963.893187] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8965.984697] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8968.667059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8968.813447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8970.907321] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 8973.486084] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8980.070844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8990.693368] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8990.694047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8990.694837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8990.698921] Lustre: Skipped 1 previous similar message [ 8995.232785] Lustre: server umount lustre-MDT0000 complete [ 8995.810058] LustreError: 439813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.816184] LustreError: 439813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8996.689685] LustreError: 440916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911951 with bad export cookie 14491297477312837711 [ 8996.694946] LustreError: 440916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8996.696241] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8996.846869] Lustre: server umount lustre-MDT0001 complete [ 9008.619996] Lustre: server umount lustre-OST0000 complete [ 9013.728288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9013.917793] LustreError: 442804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9013.947111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9015.445639] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9018.655644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9018.811959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9020.455344] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9023.169914] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9025.100189] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9027.619570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9027.761299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9029.861151] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9032.394124] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9039.013987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9054.690738] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.691930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9054.692262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.692268] Lustre: Skipped 2 previous similar messages [ 9054.696547] Lustre: Skipped 2 previous similar messages [ 9056.042709] Lustre: server umount lustre-MDT0000 complete [ 9057.553487] LustreError: 442786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912011 with bad export cookie 14491297477312838810 [ 9057.556568] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9057.558080] LustreError: 442786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9057.692523] Lustre: server umount lustre-MDT0001 complete [ 9069.420684] Lustre: server umount lustre-OST0000 complete [ 9074.956304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.140931] LustreError: 446734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.162881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9076.687266] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9079.831059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.527586] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9084.227154] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9086.182693] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9088.869928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9091.007939] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9093.592745] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9100.261956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9110.499164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9110.499833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9110.500583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9110.500588] Lustre: Skipped 2 previous similar messages [ 9110.505349] Lustre: Skipped 2 previous similar messages [ 9110.508551] LustreError: Skipped 1 previous similar message [ 9111.696757] Lustre: server umount lustre-OST0000 complete [ 9115.617239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.622668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9115.624710] Lustre: Skipped 2 previous similar messages [ 9120.736898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9120.739620] Lustre: Skipped 1 previous similar message [ 9127.391104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9127.442793] Lustre: server umount lustre-MDT0000 complete [ 9128.726773] LustreError: 446714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912083 with bad export cookie 14491297477312839818 [ 9128.729704] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9128.732075] LustreError: 446714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9128.872116] Lustre: server umount lustre-MDT0001 complete [ 9132.615355] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9133.731304] Key type lgssc unregistered [ 9133.859628] LNet: 450298:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9133.862817] LNetError: 450298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9133.875429] LNet: Removed LNI 192.168.204.115@tcp [ 9134.199234] Key type .llcrypt unregistered [ 9134.200542] Key type ._llcrypt unregistered [ 9143.793717] Key type ._llcrypt registered [ 9143.794884] Key type .llcrypt registered [ 9143.830524] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9144.384948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9144.404958] alg: No test for adler32 (adler32-zlib) [ 9145.254588] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9145.345520] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9146.927226] Key type lgssc registered [ 9147.323806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9150.233326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9150.239974] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.243625] LustreError: 451717:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9150.253016] LustreError: 451717:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9150.255971] LustreError: 451717:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9150.258285] LustreError: 451717:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9150.260732] LustreError: 451717:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9152.890412] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9152.893711] LustreError: 452113:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9152.904204] LustreError: 452113:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9152.907068] LustreError: 452113:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9152.910264] LustreError: 452113:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9152.913696] LustreError: 452113:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9158.972681] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9159.883992] Key type lgssc unregistered [ 9160.011273] LNet: 453113:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9160.013493] LNetError: 453113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9160.020423] LNet: Removed LNI 192.168.204.115@tcp [ 9160.314500] Key type .llcrypt unregistered [ 9160.315555] Key type ._llcrypt unregistered [ 9166.160925] Key type ._llcrypt registered [ 9166.162480] Key type .llcrypt registered [ 9166.201850] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9172.393470] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9172.687475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9172.696510] alg: No test for adler32 (adler32-zlib) [ 9173.551104] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9173.645912] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9175.231132] Key type lgssc registered [ 9175.564392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9186.787792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9186.793538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9187.874808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9187.886987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9187.926162] Lustre: lustre-MDT0000: new disk, initializing [ 9187.949976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9187.956436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9189.299149] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9193.449302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9193.494335] Lustre: 456240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9193.520202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9193.522228] Lustre: Skipped 1 previous similar message [ 9193.556287] Lustre: lustre-MDT0001: new disk, initializing [ 9193.590018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9193.600847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9193.604138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9194.864393] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9198.560536] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9200.276783] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9202.537135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9202.642684] Lustre: lustre-OST0000: new disk, initializing [ 9202.644259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9202.647490] Lustre: 458212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9202.677510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.600141] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9205.244244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9205.248209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9205.285204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9208.415790] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9209.807161] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9209.873558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9211.263357] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9211.348893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9215.456586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9215.458689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.459465] LustreError: Skipped 1 previous similar message [ 9215.464143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9215.468148] Lustre: Skipped 1 previous similar message [ 9218.129994] Lustre: server umount lustre-OST0000 complete [ 9219.551866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9219.554394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.558824] Lustre: Skipped 1 previous similar message [ 9219.560870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.576435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.576943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.508166] Lustre: server umount lustre-MDT0000 complete [ 9226.790358] LustreError: 457037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912181 with bad export cookie 3539386707263263452 [ 9226.793624] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9226.795610] LustreError: 457037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9226.920903] Lustre: server umount lustre-MDT0001 complete [ 9229.844563] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9230.409037] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:03:04 (1788912184) [ 9232.568338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9232.723092] LustreError: 460666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9232.730566] LustreError: 460666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9232.748262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9234.104052] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9236.906606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.024583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9238.366188] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9240.747189] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9242.741621] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9245.109684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9245.227903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9247.165612] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9249.657894] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9251.854843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9251.924547] Lustre: lustre-OST0001: new disk, initializing [ 9251.926836] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9251.930545] Lustre: 463497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9252.541050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9252.545303] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9252.584116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9253.892158] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9257.538474] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9263.076042] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9263.076653] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9263.079028] LustreError: Skipped 1 previous similar message [ 9263.084159] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9263.087250] Lustre: Skipped 2 previous similar messages [ 9265.403266] Lustre: server umount lustre-OST0001 complete [ 9268.194497] LustreError: 462485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9268.198527] LustreError: 462485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9273.315647] LustreError: 462483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9273.321396] LustreError: 462483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9277.920283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9277.922737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9277.926309] Lustre: Skipped 1 previous similar message [ 9277.927952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9277.929559] Lustre: Skipped 1 previous similar message [ 9278.433740] LustreError: 462485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9278.439953] LustreError: 462485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9283.411558] Lustre: server umount lustre-OST0000 complete [ 9288.159825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9288.162228] LustreError: Skipped 1 previous similar message [ 9288.163914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.168370] Lustre: Skipped 1 previous similar message [ 9288.170155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9288.172195] Lustre: Skipped 2 previous similar messages [ 9290.905762] Lustre: server umount lustre-MDT0000 complete [ 9292.080143] LustreError: 462506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912246 with bad export cookie 3539386707263264320 [ 9292.083096] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9292.083496] LustreError: 462506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9298.217975] Lustre: server umount lustre-MDT0001 complete [ 9301.765035] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9302.759369] Key type lgssc unregistered [ 9302.891307] LNet: 465482:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9302.894914] LNetError: 465482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9302.903264] LNet: Removed LNI 192.168.204.115@tcp [ 9303.178126] Key type .llcrypt unregistered [ 9303.179379] Key type ._llcrypt unregistered [ 9320.216403] Key type ._llcrypt registered [ 9320.218161] Key type .llcrypt registered [ 9320.253823] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9320.696833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9320.710224] alg: No test for adler32 (adler32-zlib) [ 9321.573937] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9321.674487] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9323.255170] Key type lgssc registered [ 9323.653185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9326.226329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9326.232141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.317971] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9327.324368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9327.377614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9328.693167] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9331.368386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.404321] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9331.415092] Lustre: 468255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9331.506532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9332.838189] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9335.302074] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9337.094224] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9338.848893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.849737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.853694] Lustre: Skipped 1 previous similar message [ 9343.968079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9343.970719] Lustre: Skipped 2 previous similar messages [ 9349.088086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9349.090371] Lustre: Skipped 1 previous similar message [ 9352.159072] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9352.218355] Lustre: server umount lustre-MDT0000 complete [ 9353.392090] LustreError: 468246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912307 with bad export cookie 5513437546766852089 [ 9353.394491] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9353.397180] LustreError: 468246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9353.504552] Lustre: server umount lustre-MDT0001 complete [ 9356.352433] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:05:10 (1788912310) [ 9358.376044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9358.536207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9359.782924] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9362.431411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.550139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9363.821580] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9366.081581] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9367.904534] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9370.203475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9370.262699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9370.317080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9372.097799] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9374.444297] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9378.126454] Lustre: Permanently deactivating lustre-OST0000 [ 9378.131814] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9380.850638] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9381.319931] LustreError: 473657:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9381.324511] LustreError: 473657:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9381.327386] LustreError: 473657:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9381.911827] LustreError: 473705:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9381.914165] LustreError: 473705:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9388.624110] Lustre: server umount lustre-OST0000 complete [ 9389.599309] LustreError: 473954:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 9389.601698] LustreError: 473954:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9393.631908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9393.634440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9393.638443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9399.264083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9399.266825] Lustre: Skipped 3 previous similar messages [ 9400.314633] Lustre: server umount lustre-MDT0000 complete [ 9401.569306] LustreError: 472598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912355 with bad export cookie 5513437546766852656 [ 9401.573135] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9401.573493] LustreError: 472598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9410.775519] Lustre: server umount lustre-MDT0001 complete [ 9413.686099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9415.019482] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9415.912865] LustreError: 475111:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9417.977772] Lustre: 475450:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.115@tcp [ 9418.691903] Lustre: server umount MGS complete [ 9422.495956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9422.651248] LustreError: 476173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9422.657023] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9422.657574] LustreError: 476173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9422.660272] Lustre: Skipped 1 previous similar message [ 9422.683769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9424.031529] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9426.737484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.818630] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9428.070240] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9430.323107] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9432.167787] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9432.773330] Lustre: Permanently reactivating lustre-OST0000 [ 9432.778832] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9432.780748] Lustre: Skipped 2 previous similar messages [ 9462.239107] LustreError: 478099:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9464.781811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9464.887539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9464.890218] Lustre: Skipped 1 previous similar message [ 9466.606291] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9468.803251] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9482.211623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9482.212960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9482.214706] Lustre: Skipped 1 previous similar message [ 9482.219560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9482.221232] Lustre: Skipped 1 previous similar message [ 9486.483813] Lustre: server umount lustre-OST0000 complete [ 9488.351771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9488.355774] LustreError: Skipped 1 previous similar message [ 9488.356911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9488.359709] Lustre: Skipped 1 previous similar message [ 9492.449949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9493.787579] Lustre: server umount lustre-MDT0000 complete [ 9494.992419] LustreError: 477747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912449 with bad export cookie 5513437546766853678 [ 9494.995204] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9494.996056] LustreError: 477747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9495.108576] Lustre: server umount lustre-MDT0001 complete [ 9498.481391] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9499.404601] Key type lgssc unregistered [ 9499.528358] LNet: 480989:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9499.531679] LNetError: 480989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9499.541286] LNet: Removed LNI 192.168.204.115@tcp [ 9499.836525] Key type .llcrypt unregistered [ 9499.837987] Key type ._llcrypt unregistered [ 9505.240967] Key type ._llcrypt registered [ 9505.242032] Key type .llcrypt registered [ 9505.273841] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9511.597571] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9511.993929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9512.000864] alg: No test for adler32 (adler32-zlib) [ 9512.847066] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9512.935584] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9514.519131] Key type lgssc registered [ 9514.916730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9525.822081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9525.826899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9526.905566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9526.917666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9526.949054] Lustre: lustre-MDT0000: new disk, initializing [ 9526.968059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9526.974064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9528.183756] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9532.167167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.206430] Lustre: 484118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9532.232516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9532.234316] Lustre: Skipped 1 previous similar message [ 9532.275836] Lustre: lustre-MDT0001: new disk, initializing [ 9532.312077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9532.325198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9532.328624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9533.666464] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9537.236121] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9538.898611] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9541.085830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.182574] Lustre: lustre-OST0000: new disk, initializing [ 9541.185173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9541.188390] Lustre: 486084:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9541.217601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9543.089618] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9546.651509] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9547.256493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9547.259582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9547.286473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9547.989320] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9548.058164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9549.323722] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9549.390616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9552.353172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9552.354317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.355441] LustreError: Skipped 1 previous similar message [ 9552.355969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9552.360326] Lustre: Skipped 1 previous similar message [ 9552.364781] Lustre: Skipped 1 previous similar message [ 9556.104524] Lustre: server umount lustre-OST0000 complete [ 9557.472465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.473070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9557.475767] Lustre: Skipped 1 previous similar message [ 9562.592474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9562.594095] Lustre: Skipped 2 previous similar messages [ 9563.418856] Lustre: server umount lustre-MDT0000 complete [ 9564.544031] LustreError: 484108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912518 with bad export cookie 16773026122436127986 [ 9564.546868] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9564.548245] LustreError: 484108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9564.654701] Lustre: server umount lustre-MDT0001 complete [ 9567.069696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9567.547795] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:08:41 (1788912521) [ 9569.698191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.861533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9571.130134] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9573.734033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9574.964666] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9577.114418] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9578.851748] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9581.019362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9581.139308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.142781] Lustre: Skipped 1 previous similar message [ 9582.972803] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9585.381064] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9586.026342] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9596.993187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9599.967702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9599.969696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.974249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9600.992445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9604.563202] Lustre: server umount lustre-OST0000 complete [ 9606.113640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9606.114678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9606.117970] Lustre: Skipped 2 previous similar messages [ 9606.119533] Lustre: Skipped 2 previous similar messages [ 9611.920641] Lustre: server umount lustre-MDT0000 complete [ 9613.129261] LustreError: 488523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912567 with bad export cookie 16773026122436128854 [ 9613.133326] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9613.133577] LustreError: 488523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9613.239564] Lustre: server umount lustre-MDT0001 complete [ 9616.690963] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9617.631470] Key type lgssc unregistered [ 9617.753266] LNet: 492284:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9617.755976] LNetError: 492284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9617.764364] LNet: Removed LNI 192.168.204.115@tcp [ 9618.088442] Key type .llcrypt unregistered [ 9618.089464] Key type ._llcrypt unregistered [ 9625.732336] Key type ._llcrypt registered [ 9625.733635] Key type .llcrypt registered [ 9625.772247] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9625.873256] Key type .llcrypt unregistered [ 9625.874217] Key type ._llcrypt unregistered [ 9632.480861] Key type ._llcrypt registered [ 9632.482274] Key type .llcrypt registered [ 9632.516878] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9632.935556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9632.945227] alg: No test for adler32 (adler32-zlib) [ 9633.817226] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9633.919424] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9635.495156] Key type lgssc registered [ 9635.849536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9638.312622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9638.317582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.452063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9640.686815] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9643.296613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9643.402879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9645.185396] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9647.445429] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9648.609677] LustreError: 494470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9648.616380] LustreError: 494470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9649.598968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9649.632114] LustreError: 494469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9649.636994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9649.640985] LustreError: 494469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9649.726925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9650.945173] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9659.875103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9659.875165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.883261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9659.885892] Lustre: Skipped 1 previous similar message [ 9663.949473] Lustre: server umount lustre-OST0000 complete [ 9669.089082] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9669.091245] LustreError: Skipped 1 previous similar message [ 9669.092480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.095991] Lustre: Skipped 1 previous similar message [ 9669.097775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9670.112055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9670.112532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9671.312775] Lustre: server umount lustre-MDT0000 complete [ 9672.523628] LustreError: 496575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912626 with bad export cookie 13590636835307011581 [ 9672.527581] LustreError: 496575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9672.528055] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9672.640288] Lustre: server umount lustre-MDT0001 complete [ 9676.110453] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9677.105458] Key type lgssc unregistered [ 9677.226386] LNet: 497616:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9677.229090] LNetError: 497616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9677.243269] LNet: Removed LNI 192.168.204.115@tcp [ 9677.545112] Key type .llcrypt unregistered [ 9677.546844] Key type ._llcrypt unregistered [ 9686.643930] Key type ._llcrypt registered [ 9686.645368] Key type .llcrypt registered [ 9686.683505] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9687.163153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9687.178984] alg: No test for adler32 (adler32-zlib) [ 9688.028846] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9688.113059] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9689.687179] Key type lgssc registered [ 9690.083072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9692.782714] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9692.789790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9708.255368] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9713.375323] LustreError: 498984:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.115@tcp: failed processing log, type 4: rc = -110 [ 9739.999260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9741.798152] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9744.085988] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9746.193329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9746.364079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9746.385511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9747.693269] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9750.383586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.492154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9751.682460] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9753.968341] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9755.652373] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9759.714333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9759.714732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9759.719143] Lustre: Skipped 1 previous similar message [ 9763.917025] Lustre: server umount lustre-OST0000 complete [ 9765.856049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9765.859240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9765.864559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9765.866881] Lustre: Skipped 1 previous similar message [ 9769.952722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9769.953143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9771.282735] Lustre: server umount lustre-MDT0000 complete [ 9772.637981] LustreError: 498991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912727 with bad export cookie 1781600227502029034 [ 9772.639930] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.643184] LustreError: 498991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9772.763324] Lustre: server umount lustre-MDT0001 complete [ 9776.517164] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9777.486792] Key type lgssc unregistered [ 9777.613399] LNet: 502552:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9777.615994] LNetError: 502552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9777.624260] LNet: Removed LNI 192.168.204.115@tcp [ 9777.927597] Key type .llcrypt unregistered [ 9777.929164] Key type ._llcrypt unregistered [ 9786.896313] Key type ._llcrypt registered [ 9786.898040] Key type .llcrypt registered [ 9786.937744] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9787.213856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9787.223430] alg: No test for adler32 (adler32-zlib) [ 9788.065114] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9788.149770] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9789.727141] Key type lgssc registered [ 9790.060607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9792.529396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9792.536589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9793.705290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9795.080901] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9797.825792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.948431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9799.249652] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9801.519160] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9803.336493] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9805.391795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9805.498154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9807.360690] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9809.610452] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9818.789241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9823.712233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9823.714774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9823.719702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9824.225222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9824.226093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.226962] Lustre: Skipped 1 previous similar message [ 9824.231103] Lustre: Skipped 1 previous similar message [ 9825.272531] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.15@tcp (stopping) [ 9825.274557] Lustre: Skipped 1 previous similar message [ 9827.154875] Lustre: server umount lustre-MDT0000 complete [ 9827.862324] LustreError: 503925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.15@tcp arrived at 1788912782 with bad export cookie 16226447311905286337 [ 9827.865511] LustreError: 503925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9828.440387] LustreError: 504676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912782 with bad export cookie 16226447311905285686 [ 9828.445402] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9829.344661] LustreError: 504683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9829.344848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9829.351449] LustreError: 504683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9829.353472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.361114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9833.375726] LustreError: 503943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9833.375981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9833.382257] LustreError: 503943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9834.590616] Lustre: server umount lustre-OST0000 complete [ 9837.085146] Lustre: server umount lustre-MDT0001 complete [ 9840.893912] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9842.048891] Key type lgssc unregistered [ 9842.186492] LNet: 507646:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9842.189898] LNetError: 507646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9842.196372] LNet: Removed LNI 192.168.204.115@tcp [ 9842.572270] Key type .llcrypt unregistered [ 9842.573990] Key type ._llcrypt unregistered [ 9857.296377] Key type ._llcrypt registered [ 9857.298502] Key type .llcrypt registered [ 9857.366680] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9857.992371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9858.008401] alg: No test for adler32 (adler32-zlib) [ 9859.008626] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9859.194909] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9860.863179] Key type lgssc registered [ 9861.649729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9867.152385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9867.169706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9868.473328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9871.731676] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9877.984854] LustreError: 509041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9877.998579] LustreError: 509041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9879.060352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.401768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9882.318950] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9886.404660] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9889.196506] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9893.365476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9893.635517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9897.743727] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9902.422749] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9913.130108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9920.482193] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9920.486073] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9920.494504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9923.072171] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [ 9923.553631] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9925.301932] Lustre: server umount lustre-MDT0001 complete [ 9928.673469] LustreError: 509041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.673982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9928.674684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.675547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9928.675552] Lustre: Skipped 2 previous similar messages [ 9933.726749] Lustre: server umount lustre-OST0000 complete [ 9933.793997] LustreError: 509768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9935.557852] Lustre: server umount lustre-MDT0000 complete [ 9941.612168] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [ 9942.908189] Key type lgssc unregistered [ 9943.063855] LNet: 512746:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9943.071417] LNetError: 512746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9944.102890] LNet: Removed LNI 192.168.204.115@tcp [ 9944.511430] Key type .llcrypt unregistered [ 9944.512925] Key type ._llcrypt unregistered [ 9955.746711] Key type ._llcrypt registered [ 9955.748598] Key type .llcrypt registered [ 9955.792727] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [ 9963.211341] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9963.671350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9963.730948] alg: No test for adler32 (adler32-zlib) [ 9964.622976] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9964.737726] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [ 9966.335120] Key type lgssc registered [ 9966.785787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9982.164093] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [ 9988.663712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9988.674458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9989.784709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9989.800931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9989.841511] Lustre: lustre-MDT0000: new disk, initializing [ 9989.866553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9989.874085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9991.600245] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [ 9998.643776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9998.775842] Lustre: lustre-OST0000: new disk, initializing [ 9998.779726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9998.782810] Lustre: Skipped 1 previous similar message [ 9998.786576] Lustre: 517753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9998.829044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10001.164905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10001.171530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10001.187318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10001.370658] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10006.256589] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10012.980810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.037775] Lustre: 516737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10013.070750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10013.111774] Lustre: lustre-MDT0001: new disk, initializing [10013.165303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10013.183774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10013.188179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10014.821480] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10019.187278] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10019.288056] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10020.981253] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10023.103792] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [10025.469894] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [10027.491993] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10027.492565] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10027.496511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10027.502031] Lustre: Skipped 1 previous similar message [10027.506619] Lustre: Skipped 1 previous similar message [10030.579269] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.15@tcp (stopping) [10031.195827] Lustre: server umount lustre-MDT0001 complete [10037.728542] LustreError: 517761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10037.736044] LustreError: 517761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10042.849216] LustreError: 520502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.855857] LustreError: 520502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10047.456293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10047.459595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10047.465063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10047.968266] LustreError: 516748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10053.089648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10053.090148] LustreError: 519079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10058.207185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10058.208991] LustreError: 516748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10058.277890] Lustre: server umount lustre-OST0000 complete [10065.852548] Lustre: server umount lustre-MDT0000 complete [10070.934813] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [10072.100653] Key type lgssc unregistered [10072.237341] LNet: 521513:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10072.241084] LNetError: 521513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10072.248421] LNet: Removed LNI 192.168.204.115@tcp [10072.579949] Key type .llcrypt unregistered [10072.581416] Key type ._llcrypt unregistered [10087.377102] Key type ._llcrypt registered [10087.378706] Key type .llcrypt registered [10087.432368] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [10087.552307] Key type .llcrypt unregistered [10087.553915] Key type ._llcrypt unregistered [10099.095499] Key type ._llcrypt registered [10099.098654] Key type .llcrypt registered [10099.229437] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [10100.112578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10100.276865] alg: No test for adler32 (adler32-zlib) [10101.310428] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [10101.490651] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [10103.183217] Key type lgssc registered [10104.203734] Lustre: Echo OBD driver; http://www.lustre.org/ [10130.144731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10130.175533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10131.570969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10131.618819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10131.841300] Lustre: lustre-MDT0000: new disk, initializing [10131.992604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10132.026694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10136.483509] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10145.969802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.055085] Lustre: 524864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10146.094817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10146.099981] Lustre: Skipped 1 previous similar message [10146.149569] Lustre: lustre-MDT0001: new disk, initializing [10146.213157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10146.248555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10146.254343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10150.046704] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10158.983594] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10164.186602] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10169.563982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10169.701794] Lustre: lustre-OST0000: new disk, initializing [10169.704809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10169.708372] Lustre: 526836:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10169.761493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10172.951984] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10177.036071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10177.044077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10177.100462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10177.635972] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10182.115473] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.118198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10182.119803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10182.125320] Lustre: Skipped 2 previous similar messages [10187.232188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10187.237156] Lustre: Skipped 4 previous similar messages [10187.316524] Lustre: server umount lustre-MDT0000 complete [10189.141525] LustreError: 524857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913143 with bad export cookie 656224187921860452 [10189.144965] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10189.147912] LustreError: 524857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10189.317981] Lustre: server umount lustre-MDT0001 complete [10201.616705] Lustre: server umount lustre-OST0000 complete [10207.681954] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:19:21 (1788913161) [10211.969402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10212.278835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10214.689905] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10217.441215] LustreError: 528861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10217.453652] LustreError: 528861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10219.592268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10219.802127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10222.020413] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10226.050817] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10229.053705] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10245.989946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10246.096136] Lustre: Found index 0 for lustre-OST0000, updating log [10246.190647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10249.048458] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10252.763907] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10258.472888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10268.645061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10268.651847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10268.665337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10270.754311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10270.760452] Lustre: Skipped 2 previous similar messages [10272.097690] Lustre: server umount lustre-OST0000 complete [10285.597409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.599493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10285.603268] Lustre: Skipped 2 previous similar messages [10285.605412] Lustre: Skipped 1 previous similar message [10291.756979] Lustre: server umount lustre-MDT0000 complete [10292.704709] LustreError: 529589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10292.714296] LustreError: 529589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10293.530401] LustreError: 528843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913247 with bad export cookie 656224187921861852 [10293.532841] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10293.537265] LustreError: 528843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10293.698289] Lustre: server umount lustre-MDT0001 complete [10298.400638] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:20:52 (1788913252) [10313.503803] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [10904.261476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.548957] LustreError: 535287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10904.594241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10907.585178] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10909.665614] LustreError: 535288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.263598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.493305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10916.000615] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10920.227939] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10923.130813] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10940.074956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10940.141222] LustreError: 537117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.206350] Lustre: Found index 0 for lustre-OST0000, updating log [10940.368067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10943.723650] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [10947.749804] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10950.884423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10950.890226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10950.893103] Lustre: Skipped 3 previous similar messages [10955.019720] Lustre: server umount lustre-OST0000 complete [10958.113654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.114502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10958.123830] Lustre: Skipped 1 previous similar message [10960.161314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10960.165966] Lustre: Skipped 4 previous similar messages [10963.234686] LustreError: 535288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.252087] LustreError: 535288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10963.279727] Lustre: server umount lustre-MDT0000 complete [10965.325572] LustreError: 535270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913919 with bad export cookie 656224187921862930 [10965.330084] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10965.333587] LustreError: 535270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10965.499462] Lustre: server umount lustre-MDT0001 complete [10965.792124] LNet: 1 local NIs in recovery (showing 1): 192.168.204.115@tcp [10985.836522] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [10990.306523] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11008.696363] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:32:42 (1788913962) [11016.584446] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [11017.948077] Key type lgssc unregistered [11018.101534] LNet: 541887:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11018.105638] LNetError: 541887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11073.445998] LNet: Removed LNI 192.168.204.115@tcp [11073.836616] Key type .llcrypt unregistered [11073.838304] Key type ._llcrypt unregistered [11081.663570] Key type ._llcrypt registered [11081.666157] Key type .llcrypt registered [11081.724248] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11082.103541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11082.137752] alg: No test for adler32 (adler32-zlib) [11083.027488] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [11083.137548] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [11084.735489] Key type lgssc registered [11085.279467] Lustre: Echo OBD driver; http://www.lustre.org/ [11087.443882] LNet: Added LNI 192.168.204.115@tcp100 [8/256/0/180] [11087.830529] LNet: Added LNI 192.168.204.115@tcp101 [8/256/0/180] [11088.228312] LNet: Added LNI 192.168.204.115@tcp102 [8/256/0/180] [11088.631144] LNet: Added LNI 192.168.204.115@tcp103 [8/256/0/180] [11089.029695] LNet: Added LNI 192.168.204.115@tcp104 [8/256/0/180] [11089.443843] LNet: Added LNI 192.168.204.115@tcp105 [8/256/0/180] [11089.847503] LNet: Added LNI 192.168.204.115@tcp106 [8/256/0/180] [11090.274621] LNet: Added LNI 192.168.204.115@tcp107 [8/256/0/180] [11090.678583] LNet: Added LNI 192.168.204.115@tcp108 [8/256/0/180] [11091.078953] LNet: Added LNI 192.168.204.115@tcp109 [8/256/0/180] [11091.467293] LNet: Added LNI 192.168.204.115@tcp110 [8/256/0/180] [11091.842942] LNet: Added LNI 192.168.204.115@tcp111 [8/256/0/180] [11092.209106] LNet: Added LNI 192.168.204.115@tcp112 [8/256/0/180] [11092.581399] LNet: Added LNI 192.168.204.115@tcp113 [8/256/0/180] [11092.932931] LNet: Added LNI 192.168.204.115@tcp114 [8/256/0/180] [11093.320348] LNet: Added LNI 192.168.204.115@tcp115 [8/256/0/180] [11093.708286] LNet: Added LNI 192.168.204.115@tcp116 [8/256/0/180] [11094.068550] LNet: Added LNI 192.168.204.115@tcp117 [8/256/0/180] [11094.434206] LNet: Added LNI 192.168.204.115@tcp118 [8/256/0/180] [11094.796501] LNet: Added LNI 192.168.204.115@tcp119 [8/256/0/180] [11095.202064] LNet: Added LNI 192.168.204.115@tcp120 [8/256/0/180] [11095.582397] LNet: Added LNI 192.168.204.115@tcp121 [8/256/0/180] [11095.982445] LNet: Added LNI 192.168.204.115@tcp122 [8/256/0/180] [11096.392364] LNet: Added LNI 192.168.204.115@tcp123 [8/256/0/180] [11096.857883] LNet: Added LNI 192.168.204.115@tcp124 [8/256/0/180] [11097.288269] LNet: Added LNI 192.168.204.115@tcp125 [8/256/0/180] [11097.687868] LNet: Added LNI 192.168.204.115@tcp126 [8/256/0/180] [11098.100985] LNet: Added LNI 192.168.204.115@tcp127 [8/256/0/180] [11098.514919] LNet: Added LNI 192.168.204.115@tcp128 [8/256/0/180] [11098.939637] LNet: Added LNI 192.168.204.115@tcp129 [8/256/0/180] [11099.362987] LNet: Added LNI 192.168.204.115@tcp130 [8/256/0/180] [11099.794801] LNet: Added LNI 192.168.204.115@tcp131 [8/256/0/180] [11100.244656] LNet: Added LNI 192.168.204.115@tcp132 [8/256/0/180] [11100.650835] LNet: Added LNI 192.168.204.115@tcp133 [8/256/0/180] [11101.032877] LNet: Added LNI 192.168.204.115@tcp134 [8/256/0/180] [11101.410731] LNet: Added LNI 192.168.204.115@tcp135 [8/256/0/180] [11104.349796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11104.363371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11105.417792] Lustre: 544883:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [11105.437909] LNetError: 544903:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [11105.446051] LNetError: Protocol error connecting to 192.168.204.115@tcp119 at host 192.168.204.115:988: is it running a compatible version of Lustre? [11105.452049] LNetError: Skipped 3 previous similar messages [11105.541818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11105.562707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11105.608602] Lustre: lustre-MDT0000: new disk, initializing [11105.659828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11105.674876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11105.695112] LNet: 10 local NIs in recovery (showing 10): 192.168.204.115@tcp120, 192.168.204.115@tcp121, 192.168.204.115@tcp123, 192.168.204.115@tcp126, 192.168.204.115@tcp119, 192.168.204.115@tcp122, 192.168.204.115@tcp124, 192.168.204.115@tcp125, 192.168.204.115@tcp127, 192.168.204.115@tcp128 [11106.465744] LNetError: 544903:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [11106.472169] LNetError: 544903:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [11106.476799] LNetError: Protocol error connecting to 192.168.204.115@tcp109 at host 192.168.204.115:988: is it running a compatible version of Lustre? [11106.482741] LNetError: Skipped 5 previous similar messages [11107.555613] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11108.769220] LNetError: 544903:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [11108.779097] LNetError: 544903:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [11108.784463] LNetError: Protocol error connecting to 192.168.204.115@tcp120 at host 192.168.204.115:988: is it running a compatible version of Lustre? [11108.793120] LNetError: Skipped 15 previous similar messages [11112.594829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11112.662396] Lustre: 544901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11112.700229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11112.703345] Lustre: Skipped 1 previous similar message [11112.739811] Lustre: lustre-MDT0001: new disk, initializing [11112.753491] LNetError: 542346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [11112.758434] LNetError: 542346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [11112.761070] Lustre: 545702:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [11112.761559] LNetError: Protocol error connecting to 192.168.204.115@tcp129 at host 192.168.204.115:988: is it running a compatible version of Lustre? [11112.761564] LNetError: Skipped 1 previous similar message [11112.772624] Lustre: 545702:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11112.839277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11112.868213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11112.874379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11114.720232] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11119.010115] LNetError: 545294:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [11119.015324] LNetError: 545294:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 21 previous similar messages [11119.025901] LNetError: Protocol error connecting to 192.168.204.115@tcp129 at host 192.168.204.115:988: is it running a compatible version of Lustre? [11119.031368] LNetError: Skipped 24 previous similar messages [11119.554557] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11122.072523] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11125.039709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11125.183938] Lustre: lustre-OST0000: new disk, initializing [11125.186987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11125.190653] Lustre: 546875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11125.234824] Lustre: 546880:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [11125.239267] Lustre: 546880:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [11125.267235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11127.200573] LNetError: 542347:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [11127.206981] LNetError: 542347:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [11127.210990] LNetError: Protocol error connecting to 192.168.204.115@tcp129 at host 192.168.204.115:988: is it running a compatible version of Lustre? [11127.217121] LNetError: Skipped 2 previous similar messages [11127.884471] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11132.831393] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11132.923401] Lustre: 547586:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.115@tcp: can't add NID 192.168.204.115@tcp130: rc = -75 [11132.928896] Lustre: 547586:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11132.963657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11132.969145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11133.040219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11138.017601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11138.020156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11138.021193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11138.035770] Lustre: Skipped 1 previous similar message [11142.321273] Lustre: server umount lustre-OST0000 complete [11143.584814] LNetError: 544909:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.115 [11143.589366] LNetError: 544909:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [11143.593969] LNetError: Protocol error connecting to 192.168.204.115@tcp129 at host 192.168.204.115:988: is it running a compatible version of Lustre? [11143.599710] LNetError: Skipped 5 previous similar messages [11148.258996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11148.262398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11148.273777] Lustre: Skipped 2 previous similar messages [11150.194939] Lustre: server umount lustre-MDT0000 complete [11152.547885] LustreError: 544892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914106 with bad export cookie 15884197150912439588 [11152.550773] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11152.554165] LustreError: 544892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11152.861235] Lustre: server umount lustre-MDT0001 complete [11163.391730] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [11166.112051] LNet: 3 local NIs in recovery (showing 3): 192.168.204.115@tcp129, 192.168.204.115@tcp120, 192.168.204.115@tcp128 [11166.716135] Key type lgssc unregistered [11167.269816] LNet: 548787:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11167.297287] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11167.311305] LNet: Removed LNI 192.168.204.115@tcp [11168.351790] LNet: Removed LNI 192.168.204.115@tcp100 [11168.362311] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11168.379493] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11168.390073] LNet: Removed LNI 192.168.204.115@tcp101 [11169.439649] LNet: Removed LNI 192.168.204.115@tcp102 [11169.447761] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11169.463766] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11169.476678] LNet: Removed LNI 192.168.204.115@tcp103 [11170.528183] LNet: Removed LNI 192.168.204.115@tcp104 [11171.558482] LNet: Removed LNI 192.168.204.115@tcp105 [11171.570074] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11171.581694] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11171.593601] LNet: Removed LNI 192.168.204.115@tcp106 [11172.639213] LNet: Removed LNI 192.168.204.115@tcp107 [11173.663254] LNet: Removed LNI 192.168.204.115@tcp108 [11174.691561] LNet: Removed LNI 192.168.204.115@tcp109 [11175.712426] LNet: Removed LNI 192.168.204.115@tcp110 [11175.720411] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11175.731390] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11175.736971] LNet: Removed LNI 192.168.204.115@tcp111 [11176.799673] LNet: Removed LNI 192.168.204.115@tcp112 [11177.823345] LNet: Removed LNI 192.168.204.115@tcp113 [11178.847642] LNet: Removed LNI 192.168.204.115@tcp114 [11179.871173] LNet: Removed LNI 192.168.204.115@tcp115 [11180.895241] LNet: Removed LNI 192.168.204.115@tcp116 [11181.919265] LNet: Removed LNI 192.168.204.115@tcp117 [11182.943747] LNet: Removed LNI 192.168.204.115@tcp118 [11183.971349] LNet: Removed LNI 192.168.204.115@tcp119 [11183.978619] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11183.985329] LNetError: 548787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11183.996143] LNet: Removed LNI 192.168.204.115@tcp120 [11185.055246] LNet: Removed LNI 192.168.204.115@tcp121 [11186.079231] LNet: Removed LNI 192.168.204.115@tcp122 [11187.103189] LNet: Removed LNI 192.168.204.115@tcp123 [11188.127265] LNet: Removed LNI 192.168.204.115@tcp124 [11189.151451] LNet: Removed LNI 192.168.204.115@tcp125 [11190.176540] LNet: Removed LNI 192.168.204.115@tcp126 [11191.199208] LNet: Removed LNI 192.168.204.115@tcp127 [11191.206684] LNet: Removed LNI 192.168.204.115@tcp128 [11191.218634] LNet: Removed LNI 192.168.204.115@tcp129 [11192.287346] LNet: Removed LNI 192.168.204.115@tcp130 [11193.311291] LNet: Removed LNI 192.168.204.115@tcp131 [11194.335542] LNet: Removed LNI 192.168.204.115@tcp132 [11195.359807] LNet: Removed LNI 192.168.204.115@tcp133 [11196.383675] LNet: Removed LNI 192.168.204.115@tcp134 [11197.458427] LNet: Removed LNI 192.168.204.115@tcp135 [11198.216945] Key type .llcrypt unregistered [11198.218866] Key type ._llcrypt unregistered [11231.676194] Key type ._llcrypt registered [11231.677739] Key type .llcrypt registered [11231.874995] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [11254.619860] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11257.221734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11257.320351] alg: No test for adler32 (adler32-zlib) [11258.483129] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [11258.770610] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [11260.543361] Key type lgssc registered [11261.633270] Lustre: Echo OBD driver; http://www.lustre.org/ [11304.092896] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:37:36 (1788914256) [11322.792613] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [11325.456194] Key type lgssc unregistered [11325.747869] LNet: 552884:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11325.759402] LNetError: 552884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11325.785622] LNet: Removed LNI 192.168.204.115@tcp [11326.774492] Key type .llcrypt unregistered [11326.776456] Key type ._llcrypt unregistered [11346.287556] Key type ._llcrypt registered [11346.290907] Key type .llcrypt registered [11346.423192] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11347.255356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11347.275527] alg: No test for adler32 (adler32-zlib) [11348.747669] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [11349.141230] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [11350.839268] Key type lgssc registered [11352.276768] Lustre: Echo OBD driver; http://www.lustre.org/ [11361.568447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11361.613623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.909285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11362.941913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11363.012324] Lustre: lustre-MDT0000: new disk, initializing [11363.130246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11363.155647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11367.998691] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11378.441162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.591425] Lustre: 554097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11378.679672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11378.681828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11378.683319] Lustre: Skipped 1 previous similar message [11378.754335] Lustre: lustre-MDT0001: new disk, initializing [11378.832996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11378.878261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11378.892462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11384.118472] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11393.840219] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11399.441171] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11405.860957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11406.108396] Lustre: lustre-OST0000: new disk, initializing [11406.112778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11406.118293] Lustre: 556065:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11406.198918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11412.040364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11412.051882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11412.162663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11412.177276] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11420.640666] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11424.841589] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing lnet_if_list [11430.606271] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11430.641373] LNet: Added LNI 192.168.204.115@tcp10 [8/256/0/180] [11436.257645] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11436.293195] LNet: Added LNI 192.168.204.115@tcp11 [8/256/0/180] [11441.639815] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11441.678023] LNet: Added LNI 192.168.204.115@tcp12 [8/256/0/180] [11446.992616] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11447.035710] LNet: Added LNI 192.168.204.115@tcp13 [8/256/0/180] [11452.547664] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11452.606520] LNet: Added LNI 192.168.204.115@tcp14 [8/256/0/180] [11458.026900] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11458.060900] LNet: Added LNI 192.168.204.115@tcp15 [8/256/0/180] [11462.835210] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11462.866617] LNet: Added LNI 192.168.204.115@tcp16 [8/256/0/180] [11468.928102] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11468.958968] LNet: Added LNI 192.168.204.115@tcp17 [8/256/0/180] [11474.378430] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11474.433798] LNet: Added LNI 192.168.204.115@tcp18 [8/256/0/180] [11479.887654] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11479.922641] LNet: Added LNI 192.168.204.115@tcp19 [8/256/0/180] [11486.500312] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11486.542226] LNet: Added LNI 192.168.204.115@tcp20 [8/256/0/180] [11492.055313] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11492.096049] LNet: Added LNI 192.168.204.115@tcp21 [8/256/0/180] [11499.987459] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11500.022202] LNet: Added LNI 192.168.204.115@tcp22 [8/256/0/180] [11506.227110] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11506.260259] LNet: Added LNI 192.168.204.115@tcp23 [8/256/0/180] [11514.062703] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11514.094688] LNet: Added LNI 192.168.204.115@tcp24 [8/256/0/180] [11522.855160] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11522.922314] LNet: Added LNI 192.168.204.115@tcp25 [8/256/0/180] [11535.532920] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11535.577162] LNet: Added LNI 192.168.204.115@tcp26 [8/256/0/180] [11546.083542] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11546.151291] LNet: Added LNI 192.168.204.115@tcp27 [8/256/0/180] [11552.965988] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11553.023736] LNet: Added LNI 192.168.204.115@tcp28 [8/256/0/180] [11561.687636] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11561.716029] LNet: Added LNI 192.168.204.115@tcp29 [8/256/0/180] [11568.492981] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11568.519409] LNet: Added LNI 192.168.204.115@tcp30 [8/256/0/180] [11577.374429] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11577.406119] LNet: Added LNI 192.168.204.115@tcp31 [8/256/0/180] [11588.712779] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11588.796109] LNet: Added LNI 192.168.204.115@tcp32 [8/256/0/180] [11604.998647] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11605.120284] LNet: Added LNI 192.168.204.115@tcp33 [8/256/0/180] [11618.757827] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11618.846795] LNet: Added LNI 192.168.204.115@tcp34 [8/256/0/180] [11629.237365] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11629.275487] LNet: Added LNI 192.168.204.115@tcp35 [8/256/0/180] [11637.642542] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11637.672750] LNet: Added LNI 192.168.204.115@tcp36 [8/256/0/180] [11648.374939] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11648.423088] LNet: Added LNI 192.168.204.115@tcp37 [8/256/0/180] [11660.761715] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11660.811853] LNet: Added LNI 192.168.204.115@tcp38 [8/256/0/180] [11669.817233] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11669.869572] LNet: Added LNI 192.168.204.115@tcp39 [8/256/0/180] [11678.784362] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11678.812548] LNet: Added LNI 192.168.204.115@tcp40 [8/256/0/180] [11686.723517] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11686.750254] LNet: Added LNI 192.168.204.115@tcp41 [8/256/0/180] [11695.605317] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11695.634752] LNet: Added LNI 192.168.204.115@tcp42 [8/256/0/180] [11704.536710] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11704.609812] LNet: Added LNI 192.168.204.115@tcp43 [8/256/0/180] [11714.382526] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11714.420519] LNet: Added LNI 192.168.204.115@tcp44 [8/256/0/180] [11723.534854] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11723.570804] LNet: Added LNI 192.168.204.115@tcp45 [8/256/0/180] [11731.956657] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11731.996127] LNet: Added LNI 192.168.204.115@tcp46 [8/256/0/180] [11740.037584] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11740.061086] LNet: Added LNI 192.168.204.115@tcp47 [8/256/0/180] [11748.273924] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11748.304882] LNet: Added LNI 192.168.204.115@tcp48 [8/256/0/180] [11756.755277] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11756.787728] LNet: Added LNI 192.168.204.115@tcp49 [8/256/0/180] [11765.994815] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11766.027521] LNet: Added LNI 192.168.204.115@tcp50 [8/256/0/180] [11783.656769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11783.668657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11783.687934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11786.214899] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.232698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11786.253320] Lustre: Skipped 1 previous similar message [11788.502843] Lustre: server umount lustre-MDT0000 complete [11792.420746] LustreError: 554089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914746 with bad export cookie 1853541750869828216 [11792.423648] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11792.444630] LustreError: 554089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11792.969186] Lustre: server umount lustre-MDT0001 complete [11811.755204] Lustre: 553370:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914750/real 1788914750] req@0000000047bc078e x1875812604375680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788914766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11811.788818] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11811.799636] Lustre: Skipped 1 previous similar message [11812.839461] Lustre: 553369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914751/real 1788914751] req@00000000ec7d498a x1875812604375808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788914767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11813.715968] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11816.927798] Lustre: 553368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914755/real 1788914755] req@00000000e61d1019 x1875812604375936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788914771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11818.977586] Lustre: 553370:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914757/real 1788914757] req@000000008e080f82 x1875812604376192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788914773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11823.101878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11824.095163] Lustre: 553371:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788914762/real 1788914762] req@000000000c42af11 x1875812604376320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788914778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11838.825262] LustreError: 556059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11838.861895] LustreError: 556059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11838.935211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11840.928122] LustreError: 556060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11840.936198] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11842.976253] LustreError: 556059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11844.481904] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11848.168434] LustreError: 556060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.288219] LustreError: 556059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11855.291963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11855.583808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11856.625977] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11860.504571] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [11869.166523] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11877.304461] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11905.505482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11905.531704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11905.568449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11907.050056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11911.985352] Lustre: server umount lustre-OST0000 complete [11918.312267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11918.317837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11918.346940] Lustre: Skipped 2 previous similar messages [11918.379580] Lustre: Skipped 4 previous similar messages [11924.151523] Lustre: server umount lustre-MDT0000 complete [11928.096195] LustreError: 554089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788914882 with bad export cookie 1853541750869830918 [11928.100261] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11928.118472] LustreError: 554089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11928.762214] Lustre: server umount lustre-MDT0001 complete [11943.243753] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [11946.440560] Key type lgssc unregistered [11946.888645] LNet: 583120:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11946.901602] LNetError: 583120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11946.913600] LNet: Removed LNI 192.168.204.115@tcp [11946.925778] LNet: Removed LNI 192.168.204.115@tcp10 [11946.930597] LNet: Removed LNI 192.168.204.115@tcp11 [11946.937679] LNet: Removed LNI 192.168.204.115@tcp12 [11946.943875] LNet: Removed LNI 192.168.204.115@tcp13 [11946.952268] LNet: Removed LNI 192.168.204.115@tcp14 [11946.961448] LNet: Removed LNI 192.168.204.115@tcp15 [11946.967615] LNet: Removed LNI 192.168.204.115@tcp16 [11946.971188] LNet: Removed LNI 192.168.204.115@tcp17 [11946.975261] LNet: Removed LNI 192.168.204.115@tcp18 [11946.977262] LNet: Removed LNI 192.168.204.115@tcp19 [11946.982390] LNet: Removed LNI 192.168.204.115@tcp20 [11946.989873] LNet: Removed LNI 192.168.204.115@tcp21 [11946.996246] LNet: Removed LNI 192.168.204.115@tcp22 [11947.000281] LNet: Removed LNI 192.168.204.115@tcp23 [11947.002351] LNet: Removed LNI 192.168.204.115@tcp24 [11947.009268] LNet: Removed LNI 192.168.204.115@tcp25 [11947.014188] LNet: Removed LNI 192.168.204.115@tcp26 [11947.018255] LNet: Removed LNI 192.168.204.115@tcp27 [11947.021885] LNet: Removed LNI 192.168.204.115@tcp28 [11947.037814] LNet: Removed LNI 192.168.204.115@tcp29 [11947.044953] LNet: Removed LNI 192.168.204.115@tcp30 [11947.050880] LNet: Removed LNI 192.168.204.115@tcp31 [11947.054880] LNet: Removed LNI 192.168.204.115@tcp32 [11947.056770] LNet: Removed LNI 192.168.204.115@tcp33 [11947.064901] LNet: Removed LNI 192.168.204.115@tcp34 [11947.071415] LNet: Removed LNI 192.168.204.115@tcp35 [11947.072935] LNet: Removed LNI 192.168.204.115@tcp36 [11947.076432] LNet: Removed LNI 192.168.204.115@tcp37 [11947.078132] LNet: Removed LNI 192.168.204.115@tcp38 [11947.081538] LNet: Removed LNI 192.168.204.115@tcp39 [11947.084461] LNet: Removed LNI 192.168.204.115@tcp40 [11947.087375] LNet: Removed LNI 192.168.204.115@tcp41 [11947.091251] LNet: Removed LNI 192.168.204.115@tcp42 [11947.094454] LNet: Removed LNI 192.168.204.115@tcp43 [11947.096525] LNet: Removed LNI 192.168.204.115@tcp44 [11947.099681] LNet: Removed LNI 192.168.204.115@tcp45 [11947.101997] LNet: Removed LNI 192.168.204.115@tcp46 [11947.104184] LNet: Removed LNI 192.168.204.115@tcp47 [11947.106819] LNet: Removed LNI 192.168.204.115@tcp48 [11947.108898] LNet: Removed LNI 192.168.204.115@tcp49 [11947.121845] LNet: Removed LNI 192.168.204.115@tcp50 [11948.115447] Key type .llcrypt unregistered [11948.118043] Key type ._llcrypt unregistered [11969.746419] Key type ._llcrypt registered [11969.748912] Key type .llcrypt registered [11969.871160] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [11985.176945] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [11986.079627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11986.130241] alg: No test for adler32 (adler32-zlib) [11987.390719] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [11987.637364] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [11989.375211] Key type lgssc registered [11990.676684] Lustre: Echo OBD driver; http://www.lustre.org/ [12028.902406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12028.928175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12030.257702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12030.348120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12030.429221] Lustre: lustre-MDT0000: new disk, initializing [12030.518781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12030.535621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12035.465347] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12047.293933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.434263] Lustre: 586319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12047.488383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12047.496583] Lustre: Skipped 1 previous similar message [12047.655970] Lustre: lustre-MDT0001: new disk, initializing [12047.786443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12047.854130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12047.866487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12053.354712] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12064.770583] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12072.062960] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12080.978212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12081.267382] Lustre: *** cfs_fail_loc=90f, val=0*** [12081.394223] Lustre: lustre-OST0000: new disk, initializing [12081.398727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12081.410863] Lustre: 588337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12081.541347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12088.441263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12088.457669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12088.596160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12088.828801] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12099.750549] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12109.294815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12109.297384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12109.317938] Lustre: Skipped 1 previous similar message [12109.320324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12114.247549] Lustre: server umount lustre-OST0000 complete [12119.522994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.526955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12119.539935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12119.540034] Lustre: Skipped 1 previous similar message [12119.563063] LustreError: Skipped 1 previous similar message [12124.681380] Lustre: server umount lustre-MDT0000 complete [12128.988847] LustreError: 586311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915083 with bad export cookie 15489933517919262722 [12128.990893] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12129.006966] LustreError: 586311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12129.309649] Lustre: server umount lustre-MDT0001 complete [12140.481814] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12144.327537] Key type lgssc unregistered [12144.664405] LNet: 590244:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12144.674297] LNetError: 590244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12144.704476] LNet: Removed LNI 192.168.204.115@tcp [12145.942431] Key type .llcrypt unregistered [12145.944790] Key type ._llcrypt unregistered [12169.421841] Key type ._llcrypt registered [12169.424716] Key type .llcrypt registered [12169.617746] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12171.127028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12171.167952] alg: No test for adler32 (adler32-zlib) [12172.212228] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [12172.406603] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [12174.063207] Key type lgssc registered [12175.298320] Lustre: Echo OBD driver; http://www.lustre.org/ [12183.695001] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12183.724882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12185.159388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12189.917962] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12195.298094] LustreError: 591514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.327071] LustreError: 591514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12200.287048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12200.419705] LustreError: 591515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.807061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12206.102992] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12215.121946] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12222.238270] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12229.667875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12230.041344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12236.803812] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12246.458770] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12258.689724] loop0: detected capacity change from 0 to 204800000 [12259.009758] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12259.032903] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12259.057506] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12265.901981] loop0: detected capacity change from 0 to 204800000 [12266.145737] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12266.175273] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12266.187141] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12273.418204] loop0: detected capacity change from 0 to 204800000 [12280.290253] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12280.609611] Lustre: fs15246-OST0000: new disk, initializing [12280.614422] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12280.624247] Lustre: 595112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12280.758924] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12288.711552] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12300.271225] loop1: detected capacity change from 0 to 204800000 [12306.618960] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12306.774940] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12306.865480] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12306.982386] Lustre: fs15246-MDT0000: new disk, initializing [12307.136658] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12307.202988] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12307.331936] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12308.384567] Lustre: 596237:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000022188b41 x1875813467210752/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788915278 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12311.829861] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12317.476386] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12317.496984] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12317.521566] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12340.206926] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12340.227212] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12341.949145] Lustre: server umount fs15246-MDT0000 complete [12345.320630] LustreError: 591511:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12350.451875] LustreError: 594007:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.566843] Lustre: server umount fs15246-OST0000 complete [12365.798808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12365.802110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12365.816946] Lustre: Skipped 1 previous similar message [12365.834033] Lustre: Skipped 1 previous similar message [12370.854952] Lustre: server umount lustre-OST0000 complete [12375.008359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12375.017368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.044728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12381.157388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12381.162064] Lustre: Skipped 3 previous similar messages [12386.285328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12386.300764] Lustre: Skipped 1 previous similar message [12389.343154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12389.624177] Lustre: server umount lustre-MDT0000 complete [12391.396408] LustreError: 591515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12391.430361] LustreError: 591515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12393.728762] LustreError: 591495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915348 with bad export cookie 5807192507615884361 [12393.730731] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12393.739071] LustreError: 591495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12394.154087] Lustre: server umount lustre-MDT0001 complete [12407.671520] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12410.679596] Key type lgssc unregistered [12411.062807] LNet: 599337:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12411.071639] LNetError: 599337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12411.093990] LNet: Removed LNI 192.168.204.115@tcp [12412.198330] Key type .llcrypt unregistered [12412.200234] Key type ._llcrypt unregistered [12434.237497] Key type ._llcrypt registered [12434.243514] Key type .llcrypt registered [12434.354363] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12436.376832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12436.436528] alg: No test for adler32 (adler32-zlib) [12437.719805] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [12438.193086] LNet: Added LNI 192.168.204.115@tcp [8/256/0/180] [12439.976955] Key type lgssc registered [12442.271760] Lustre: Echo OBD driver; http://www.lustre.org/ [12491.656600] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [12504.864299] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12548.956942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12549.026060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12550.512830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12550.552577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12550.690748] Lustre: lustre-MDT0000: new disk, initializing [12550.889929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12550.930202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12557.974883] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12570.429803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12570.530869] Lustre: 603807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12570.630647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12570.636432] Lustre: Skipped 1 previous similar message [12570.811204] Lustre: lustre-MDT0001: new disk, initializing [12570.921802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12570.968480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12570.976161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12575.602091] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12587.477201] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12595.019905] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12604.197542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12604.727862] Lustre: lustre-OST0000: new disk, initializing [12604.732255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12604.744488] Lustre: 605779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12604.869871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12612.680728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12612.696353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12612.921096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12613.771709] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_default_debug -1 all [12626.611894] Lustre: DEBUG MARKER: oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12632.294325] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12632.548226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12638.678421] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12638.950985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12643.811599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12643.821194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12643.824782] LustreError: Skipped 1 previous similar message [12643.825392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12643.888072] Lustre: Skipped 1 previous similar message [12647.516828] Lustre: server umount lustre-OST0000 complete [12653.029544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12653.039325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.063675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12653.073666] Lustre: Skipped 1 previous similar message [12654.057211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12654.075197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12654.088256] Lustre: Skipped 1 previous similar message [12659.176094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12659.187785] Lustre: Skipped 1 previous similar message [12664.290682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12664.296506] Lustre: Skipped 1 previous similar message [12666.335348] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12666.642964] Lustre: server umount lustre-MDT0000 complete [12670.381908] LustreError: 603799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788915624 with bad export cookie 6380216731391390993 [12670.388338] LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12670.400085] LustreError: 603799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12670.733887] Lustre: server umount lustre-MDT0001 complete [12701.250458] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing set_hostid [12715.078562] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing load_modules_local [12756.555435] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12485 sec ========= 21:01:48 (1788915708) [12759.053445] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:01:51 (1788915711) === [12764.616399] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:01:57 (1788915717) === [12818.664976] Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local [12822.273808] Key type lgssc unregistered [12822.788865] LNet: 612442:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12822.795200] LNetError: 612442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12822.829489] LNet: Removed LNI 192.168.204.115@tcp [12824.038763] Key type .llcrypt unregistered [12824.041225] Key type ._llcrypt unregistered