[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 464120192 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003462] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008581] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009030] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012161] LSM: Security Framework initializing [ 0.013092] Yama: becoming mindful. [ 0.014254] SELinux: Initializing. [ 0.015086] *** VALIDATE selinux *** [ 0.027043] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032028] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035138] *** VALIDATE tmpfs *** [ 0.037205] *** VALIDATE proc *** [ 0.038323] *** VALIDATE cgroup *** [ 0.039009] *** VALIDATE cgroup2 *** [ 0.041131] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042292] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044039] Spectre V2 : User space: Vulnerable [ 0.045010] Speculative Store Bypass: Vulnerable [ 0.048861] debug: unmapping init [mem 0xffffffffbc459000-0xffffffffbc460fff] [ 0.051297] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053522] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054034] ... version: 2 [ 0.055013] ... bit width: 48 [ 0.056011] ... generic registers: 4 [ 0.057014] ... value mask: 0000ffffffffffff [ 0.058016] ... max period: 00007fffffffffff [ 0.059014] ... fixed-purpose events: 3 [ 0.060013] ... event mask: 000000070000000f [ 0.062349] rcu: Hierarchical SRCU implementation. [ 0.066352] smp: Bringing up secondary CPUs ... [ 0.068327] x86: Booting SMP configuration: [ 0.069018] .... node #0, CPUs: #1 #2 #3 [ 0.075235] smp: Brought up 1 node, 4 CPUs [ 0.077024] smpboot: Max logical packages: 1 [ 0.078015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.254029] node 0 deferred pages initialised in 174ms [ 0.262008] devtmpfs: initialized [ 0.263177] x86/mm: Memory block size: 128MB [ 0.265762] gcov: version magic: 0x41383552 [ 0.267301] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.268062] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269273] pinctrl core: initialized pinctrl subsystem [ 0.270135] [ 0.270586] ************************************************************* [ 0.273009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275007] ** ** [ 0.277007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.278006] ** ** [ 0.280007] ** This means that this kernel is built to expose internal ** [ 0.281006] ** IOMMU data structures, which may compromise security on ** [ 0.283008] ** your system. ** [ 0.284007] ** ** [ 0.286007] ** If you see this message and you are not debugging the ** [ 0.288007] ** kernel, report this immediately to your vendor! ** [ 0.289006] ** ** [ 0.291008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292005] ************************************************************* [ 0.293749] NET: Registered protocol family 16 [ 0.295425] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.297038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.299051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.302040] cpuidle: using governor menu [ 0.303877] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.306422] PCI: Using configuration type 1 for base access [ 0.308123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.317050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.322087] cryptd: max_cpu_qlen set to 1000 [ 0.323815] ACPI: Added _OSI(Module Device) [ 0.324084] ACPI: Added _OSI(Processor Device) [ 0.326010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.327008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.333021] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.340654] ACPI: Interpreter enabled [ 0.341052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.342010] ACPI: Using IOAPIC for interrupt routing [ 0.343120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.344527] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.356558] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.359027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.361012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.364092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.369462] acpiphp: Slot [2] registered [ 0.371107] acpiphp: Slot [5] registered [ 0.373092] acpiphp: Slot [6] registered [ 0.374095] acpiphp: Slot [7] registered [ 0.375091] acpiphp: Slot [8] registered [ 0.377122] acpiphp: Slot [9] registered [ 0.379132] acpiphp: Slot [10] registered [ 0.380087] acpiphp: Slot [3] registered [ 0.382088] acpiphp: Slot [4] registered [ 0.383092] acpiphp: Slot [11] registered [ 0.385083] acpiphp: Slot [12] registered [ 0.386069] acpiphp: Slot [13] registered [ 0.388100] acpiphp: Slot [14] registered [ 0.389068] acpiphp: Slot [15] registered [ 0.391071] acpiphp: Slot [16] registered [ 0.392061] acpiphp: Slot [17] registered [ 0.393077] acpiphp: Slot [18] registered [ 0.395090] acpiphp: Slot [19] registered [ 0.397101] acpiphp: Slot [20] registered [ 0.398077] acpiphp: Slot [21] registered [ 0.399065] acpiphp: Slot [22] registered [ 0.400078] acpiphp: Slot [23] registered [ 0.401090] acpiphp: Slot [24] registered [ 0.403064] acpiphp: Slot [25] registered [ 0.404065] acpiphp: Slot [26] registered [ 0.405099] acpiphp: Slot [27] registered [ 0.407078] acpiphp: Slot [28] registered [ 0.409217] acpiphp: Slot [29] registered [ 0.410128] acpiphp: Slot [30] registered [ 0.412064] acpiphp: Slot [31] registered [ 0.413049] PCI host bridge to bus 0000:00 [ 0.414014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.417017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.420029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.423032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.425027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.428030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429268] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432107] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435277] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.443000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.448049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.450015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.452016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.454009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.455514] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.457576] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.460047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.463803] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.469013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.482021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.487027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.496423] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.506023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.514033] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.531016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.539943] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.552016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.566022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.579939] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.591015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.601017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.619022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.634000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.684071] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.692015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.722014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.732237] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.737015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.744016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.757020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.770412] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.778114] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.789017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.805017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.818346] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.821501] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.823549] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.826392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.828200] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.831138] iommu: Default domain type: Passthrough [ 0.832428] SCSI subsystem initialized [ 0.833114] ACPI: bus type USB registered [ 0.835130] usbcore: registered new interface driver usbfs [ 0.836080] usbcore: registered new interface driver hub [ 0.837081] usbcore: registered new device driver usb [ 0.839205] pps_core: LinuxPPS API ver. 1 registered [ 0.841032] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.843108] PTP clock support registered [ 0.845123] EDAC MC: Ver: 3.0.0 [ 0.847184] PCI: Using ACPI for IRQ routing [ 0.849018] NetLabel: Initializing [ 0.850010] NetLabel: domain hash size = 128 [ 0.851008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.852087] NetLabel: unlabeled traffic allowed by default [ 0.864158] vgaarb: loaded [ 0.897399] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.900013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.906343] clocksource: Switched to clocksource kvm-clock [ 1.029725] VFS: Disk quotas dquot_6.6.0 [ 1.031368] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.033824] *** VALIDATE ramfs *** [ 1.034955] *** VALIDATE hugetlbfs *** [ 1.036345] pnp: PnP ACPI init [ 1.038798] pnp: PnP ACPI: found 6 devices [ 1.054893] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.060231] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.062402] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.064786] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.066714] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.068948] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.071401] NET: Registered protocol family 2 [ 1.073743] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.079350] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.082372] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.087325] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.090396] TCP: Hash tables configured (established 65536 bind 65536) [ 1.092964] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.096842] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.099269] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.101993] NET: Registered protocol family 1 [ 1.104310] RPC: Registered named UNIX socket transport module. [ 1.105960] RPC: Registered udp transport module. [ 1.107653] RPC: Registered tcp transport module. [ 1.109103] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.110888] NET: Registered protocol family 44 [ 1.112536] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.114412] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.116503] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.118638] PCI: CLS 0 bytes, default 64 [ 1.120835] Unpacking initramfs... [ 3.070579] debug: unmapping init [mem 0xffff9ea8fcc54000-0xffff9ea8fffbffff] [ 3.075086] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.079562] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.082308] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.782809] Initialise system trusted keyrings [ 3.869800] Key type blacklist registered [ 3.872504] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.892448] zbud: loaded [ 3.904237] *** VALIDATE nfs *** [ 3.905479] *** VALIDATE nfs4 *** [ 3.909139] pstore: using deflate compression [ 3.942624] Platform Keyring initialized [ 4.065725] NET: Registered protocol family 38 [ 4.069418] Key type asymmetric registered [ 4.071703] Asymmetric key parser 'x509' registered [ 4.076411] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.082771] io scheduler mq-deadline registered [ 4.086738] io scheduler kyber registered [ 4.090974] io scheduler bfq registered [ 4.095636] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.099151] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.105589] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.108271] ACPI: Power Button [PWRF] [ 4.113461] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.120608] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.135928] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.152647] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.176699] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.204557] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.240745] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.245793] Non-volatile memory driver v1.3 [ 4.247141] Linux agpgart interface v0.103 [ 4.274905] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 4.279436] vda: detected capacity change from 0 to 75096064 [ 4.305132] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.307538] vdb: detected capacity change from 0 to 1073741824 [ 4.337060] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.339406] vdc: detected capacity change from 0 to 2621440000 [ 4.363422] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.366671] vdd: detected capacity change from 0 to 2621440000 [ 4.395639] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.397284] vde: detected capacity change from 0 to 4294967296 [ 4.430917] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.434100] vdf: detected capacity change from 0 to 4294967296 [ 4.455933] libphy: Fixed MDIO Bus: probed [ 4.466954] usbcore: registered new interface driver usbserial_generic [ 4.468514] usbserial: USB Serial support registered for generic [ 4.470023] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.476203] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.477682] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.479463] mousedev: PS/2 mouse device common for all mice [ 4.484550] rtc_cmos 00:05: RTC can wake from S4 [ 4.492543] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.496032] rtc_cmos 00:05: registered as rtc0 [ 4.496900] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.499995] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.501652] intel_pstate: CPU model not supported [ 4.508342] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.514917] hid: raw HID events driver (C) Jiri Kosina [ 4.520799] usbcore: registered new interface driver usbhid [ 4.522371] usbhid: USB HID core driver [ 4.523927] drop_monitor: Initializing network drop monitor service [ 4.527278] Initializing XFRM netlink socket [ 4.530905] NET: Registered protocol family 10 [ 4.533831] Segment Routing with IPv6 [ 4.537402] NET: Registered protocol family 17 [ 4.539383] mpls_gso: MPLS GSO support [ 4.564051] RAS: Correctable Errors collector initialized. [ 4.566164] AVX version of gcm_enc/dec engaged. [ 4.577162] AES CTR mode by8 optimization enabled [ 4.712225] sched_clock: Marking stable (4712153320, 0)->(5854807180, -1142653860) [ 4.730607] registered taskstats version 1 [ 4.732756] Loading compiled-in X.509 certificates [ 4.734970] zswap: loaded using pool lzo/zbud [ 4.757852] Key type big_key registered [ 4.767508] Key type encrypted registered [ 4.768644] ima: No TPM chip found, activating TPM-bypass! [ 4.770143] ima: Allocated hash algorithm: sha1 [ 4.771857] ima: No architecture policies found [ 4.773483] evm: Initialising EVM extended attributes: [ 4.775374] evm: security.selinux [ 4.776737] evm: security.ima [ 4.777770] evm: security.capability [ 4.778751] evm: HMAC attrs: 0x1 [ 4.781115] rtc_cmos 00:05: setting system clock to 2026-09-08 15:28:13 UTC (1788881293) [ 4.788259] debug: unmapping init [mem 0xffffffffbd403000-0xffffffffbd5fffff] [ 4.792153] debug: unmapping init [mem 0xffffffffbc182000-0xffffffffbc458fff] [ 4.799079] Write protecting the kernel read-only data: 28672k [ 4.802711] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 4.805465] debug: unmapping init [mem 0xffffffffbb114000-0xffffffffbb1fffff] [ 4.836654] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.842232] systemd[1]: Detected virtualization kvm. [ 4.843449] systemd[1]: Detected architecture x86-64. [ 4.844642] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.869757] systemd[1]: No hostname configured. [ 4.870979] systemd[1]: Set hostname to . [ 4.873824] random: systemd: uninitialized urandom read (16 bytes read) [ 4.877468] systemd[1]: Initializing machine ID from random generator. [ 4.956079] random: ln: uninitialized urandom read (6 bytes read) [ 5.084832] random: systemd: uninitialized urandom read (16 bytes read) [ 5.088269] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.112979] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.122919] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.208461] device-mapper: uevent: version 1.0.3 [ 6.210466] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.535961] scsi host0: ata_piix [ 7.539126] scsi host1: ata_piix [ 7.541886] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.544761] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.705058] random: fast init done [ 7.806401] virtio_net virtio0 ens2: renamed from eth0 [ 13.290455] random: crng init done [ 13.292083] random: 7 urandom warning(s) missed due to ratelimiting [ 16.004107] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.422703] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.274388] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.357135] SELinux: Disabled at runtime. [ 23.497290] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.546174] systemd[1]: Detected virtualization kvm. [ 23.563462] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.660316] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.668029] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.686715] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.695843] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.702748] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.732339] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.787184] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 28.513944] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 28.553423] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 28.631477] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 28.678365] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ 28.696717] systemd[1]: Reached target RPC Port Mapper. [ OK ] Reached target RPC Port Mapper. [ 28.730094] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 28.773365] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 28.790693] systemd[1]: Starting udev Coldplug all Devices... Starting udev Coldplug all Devices... [ 28.805390] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ 28.849973] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 28.875547] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ 31.640816] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.497779] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.665166] input: PC Speaker as /devices/platform/pcspkr/input/input5 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit)[ 39.206862] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 39.263317] EDAC sbridge: Ver: 1.1.2 [*** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit)[ 40.164915] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (13s / no limit)[ 40.770208] NFS: Registering the id_resolver key type [ 40.772765] Key type id_resolver registered [ 40.775326] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg159-server login: [ 76.533625] hrtimer: interrupt took 3610893 ns [ 128.349914] libcfs: loading out-of-tree module taints kernel. [ 128.374843] Key type ._llcrypt registered [ 128.376948] Key type .llcrypt registered [ 128.498675] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 145.747126] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 147.209888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 147.237168] alg: No test for adler32 (adler32-zlib) [ 148.528212] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 149.388429] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 151.200505] Key type lgssc registered [ 152.739984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 166.737338] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 203.183457] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 214.638617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 214.709721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.944948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.977709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.076228] Lustre: lustre-MDT0000: new disk, initializing [ 216.145249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.161173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.354407] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.435531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.532178] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.560326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 231.564332] Lustre: Skipped 1 previous similar message [ 231.633338] Lustre: lustre-MDT0001: new disk, initializing [ 231.703313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.733955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 231.748970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.354099] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.732399] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 247.221235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.471434] Lustre: lustre-OST0000: new disk, initializing [ 247.476368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 247.483130] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.518275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.451299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 250.462213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 250.564683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.482755] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.325311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.477415] Lustre: lustre-OST0001: new disk, initializing [ 264.479297] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.484149] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.521435] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.909879] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.348168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 270.359165] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.438189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 281.176395] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 292.760036] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 299.236482] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing check_logdir /tmp/testlogs/ [ 304.809594] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing yml_node [ 308.630338] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 311.410951] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 313.706499] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 315.105653] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 11:33:22 EDT 2026 [ 328.726832] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 329.781149] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 344.544789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 344.553536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.570192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.104525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.105239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.116342] Lustre: Skipped 1 previous similar message [ 347.119187] Lustre: Skipped 1 previous similar message [ 349.839104] Lustre: server umount lustre-MDT0000 complete [ 352.552535] LustreError: 9519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881641 with bad export cookie 4109256059987290112 [ 352.560627] LustreError: 9519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 352.564205] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.343769] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.345819] LustreError: 10095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.346027] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.346031] Lustre: Skipped 2 previous similar messages [ 357.368820] Lustre: Skipped 2 previous similar messages [ 357.392255] LustreError: 10095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 358.881837] Lustre: server umount lustre-MDT0001 complete [ 362.272800] Lustre: server umount lustre-OST0000 complete [ 365.351473] Lustre: server umount lustre-OST0001 complete [ 370.602531] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 378.483878] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 405.349459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.473949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 405.503187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.555238] Lustre: lustre-MDT0000: new disk, initializing [ 405.605121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.619628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 407.984364] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 415.081562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.142356] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 415.189813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 415.194519] Lustre: Skipped 1 previous similar message [ 415.232315] Lustre: lustre-MDT0001: new disk, initializing [ 415.287052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 415.314499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 415.326964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.297351] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 422.587395] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 426.329904] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 430.500593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 430.676254] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 431.832518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.838728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.961750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 434.707253] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 441.299439] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 444.205958] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.362805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 447.618210] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 447.765675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 452.579900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.582310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.584561] LustreError: Skipped 1 previous similar message [ 452.591483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.594917] Lustre: Skipped 2 previous similar messages [ 457.698067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.702842] Lustre: Skipped 1 previous similar message [ 463.327139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 463.395511] Lustre: server umount lustre-OST0000 complete [ 466.403634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.413908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.420217] Lustre: Skipped 1 previous similar message [ 466.423835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.426786] Lustre: Skipped 2 previous similar messages [ 471.287311] Lustre: server umount lustre-MDT0000 complete [ 473.056561] LustreError: 15082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.064064] LustreError: 15082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 473.188034] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881761 with bad export cookie 4109256059987291645 [ 473.189434] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 473.199815] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 473.401881] Lustre: server umount lustre-MDT0001 complete [ 475.281354] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 11:36:03 (1788881763) [ 479.324519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.657662] LustreError: 19301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.707398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.717050] Lustre: Skipped 1 previous similar message [ 482.728486] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 484.840412] LustreError: 19302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.529082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.844474] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 496.332566] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.019495] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 504.420704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 508.658621] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 512.687742] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 523.750767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 523.754911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.763621] Lustre: Skipped 1 previous similar message [ 523.773392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.783922] Lustre: Skipped 2 previous similar messages [ 527.408735] Lustre: server umount lustre-OST0000 complete [ 536.343825] Lustre: server umount lustre-MDT0000 complete [ 538.714753] LustreError: 20039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881827 with bad export cookie 4109256059987292527 [ 538.716321] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.723745] LustreError: 20039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 538.963421] Lustre: server umount lustre-MDT0001 complete [ 545.971853] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 547.936222] Key type lgssc unregistered [ 548.153673] LNet: 22828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.157902] LNetError: 22828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.169210] LNet: Removed LNI 192.168.201.159@tcp [ 548.835271] Key type .llcrypt unregistered [ 548.838243] Key type ._llcrypt unregistered [ 563.586544] Key type ._llcrypt registered [ 563.588540] Key type .llcrypt registered [ 563.679573] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 564.268819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 564.309716] alg: No test for adler32 (adler32-zlib) [ 565.216800] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 565.358897] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 566.967417] Key type lgssc registered [ 567.654293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 572.285467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 572.303083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.604582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.469808] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 581.927390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.131964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 584.366982] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 589.036360] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 592.750916] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 597.115615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 597.473380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.519613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 602.058208] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 607.560581] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 623.075109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.076378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 623.077391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 623.087230] Lustre: Skipped 1 previous similar message [ 625.536846] Lustre: server umount lustre-OST0000 complete [ 628.193035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.195790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 628.199958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.199988] Lustre: Skipped 1 previous similar message [ 628.201249] Lustre: Skipped 1 previous similar message [ 633.316486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.335761] Lustre: Skipped 2 previous similar messages [ 634.265380] Lustre: server umount lustre-MDT0000 complete [ 636.508455] LustreError: 26055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881925 with bad export cookie 2353638538227765913 [ 636.509644] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.525104] LustreError: 26055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 636.710716] Lustre: server umount lustre-MDT0001 complete [ 642.393545] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 644.212282] Key type lgssc unregistered [ 644.446662] LNet: 28183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.449904] LNetError: 28183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.478055] LNet: Removed LNI 192.168.201.159@tcp [ 646.058385] Key type .llcrypt unregistered [ 646.060248] Key type ._llcrypt unregistered [ 658.800680] Key type ._llcrypt registered [ 658.802613] Key type .llcrypt registered [ 658.862314] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 659.466560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.501337] alg: No test for adler32 (adler32-zlib) [ 660.396382] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 660.533163] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 662.127203] Key type lgssc registered [ 662.553309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 665.661532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 665.668726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.831710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.386750] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 671.947638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.115871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 673.736860] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 676.548467] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 678.703828] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 683.823511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 683.967315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 686.215747] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 689.064842] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 692.202123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 702.432141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 702.437500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.449974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 706.528690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 706.536206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.538171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 706.547908] Lustre: Skipped 1 previous similar message [ 708.248018] Lustre: server umount lustre-OST0000 complete [ 711.650385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.656186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 711.658331] Lustre: Skipped 1 previous similar message [ 716.093539] Lustre: server umount lustre-MDT0000 complete [ 716.768182] LustreError: 29572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.776175] LustreError: 29572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 717.772454] LustreError: 29554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882006 with bad export cookie 13094946265017490216 [ 717.773142] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 717.776251] LustreError: 29554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 717.896173] Lustre: server umount lustre-MDT0001 complete [ 722.221467] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 723.430588] Key type lgssc unregistered [ 723.569409] LNet: 33464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.573043] LNetError: 33464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.581360] LNet: Removed LNI 192.168.201.159@tcp [ 723.960498] Key type .llcrypt unregistered [ 723.962150] Key type ._llcrypt unregistered [ 733.584480] Key type ._llcrypt registered [ 733.585874] Key type .llcrypt registered [ 733.629146] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 734.050954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.068870] alg: No test for adler32 (adler32-zlib) [ 734.933859] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 735.030954] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 736.615162] Key type lgssc registered [ 737.020066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 739.957398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 739.966780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.143632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 742.627081] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 745.699053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.829131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 747.207040] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 749.731684] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 751.629975] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 753.929791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 754.065205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.075054] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 758.607957] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 765.350843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 775.649711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 775.650140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.653044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 775.657233] Lustre: Skipped 1 previous similar message [ 778.061832] Lustre: server umount lustre-OST0000 complete [ 780.768648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.769170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 780.772944] Lustre: Skipped 1 previous similar message [ 780.775471] Lustre: Skipped 1 previous similar message [ 785.494587] Lustre: server umount lustre-MDT0000 complete [ 785.888935] LustreError: 34848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 785.895403] LustreError: 34848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 786.754087] LustreError: 34830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882075 with bad export cookie 14343755449582474067 [ 786.757076] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.758803] LustreError: 34830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 786.867559] Lustre: server umount lustre-MDT0001 complete [ 790.424085] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 791.485159] Key type lgssc unregistered [ 791.602405] LNet: 38396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.604752] LNetError: 38396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.614351] LNet: Removed LNI 192.168.201.159@tcp [ 791.921767] Key type .llcrypt unregistered [ 791.923536] Key type ._llcrypt unregistered [ 800.366175] Key type ._llcrypt registered [ 800.367158] Key type .llcrypt registered [ 800.403130] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 800.809322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 800.818345] alg: No test for adler32 (adler32-zlib) [ 801.680540] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 801.771773] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 803.351156] Key type lgssc registered [ 803.721925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 806.399532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 806.406194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.544225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.882054] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 811.731457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.853881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 813.198290] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 815.610244] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 817.476480] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 819.811876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.927980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 821.893423] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 824.424163] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 833.254181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 838.628399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.630759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.631824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 838.634610] Lustre: Skipped 1 previous similar message [ 840.890978] Lustre: lustre-OST0000: Not available for connect from 192.168.201.59@tcp (stopping) [ 840.894329] Lustre: Skipped 1 previous similar message [ 843.745028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 843.748106] Lustre: Skipped 1 previous similar message [ 844.172569] Lustre: server umount lustre-OST0000 complete [ 857.223689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.229332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 857.231029] Lustre: Skipped 1 previous similar message [ 862.688157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 862.690783] Lustre: Skipped 1 previous similar message [ 863.446347] Lustre: server umount lustre-MDT0000 complete [ 864.771156] LustreError: 39763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882153 with bad export cookie 18307356866726095344 [ 864.774447] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 864.775445] LustreError: 39763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 879.071113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 879.155295] Lustre: server umount lustre-MDT0001 complete [ 882.698479] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 883.700325] Key type lgssc unregistered [ 883.834410] LNet: 43505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.836721] LNetError: 43505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.847296] LNet: Removed LNI 192.168.201.159@tcp [ 884.141849] Key type .llcrypt unregistered [ 884.142935] Key type ._llcrypt unregistered [ 892.802932] Key type ._llcrypt registered [ 892.804248] Key type .llcrypt registered [ 892.842505] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 893.120072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.131905] alg: No test for adler32 (adler32-zlib) [ 893.987050] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 894.078077] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 895.663120] Key type lgssc registered [ 896.027385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 898.583878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 898.590803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.733144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.065680] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 903.853041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 903.979096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 905.353428] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 907.883948] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.743930] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 911.948069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 912.061719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.966726] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 916.348117] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 925.413254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 929.759872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 929.762497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.768821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.786543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.787368] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.788505] Lustre: Skipped 1 previous similar message [ 930.791806] Lustre: Skipped 1 previous similar message [ 932.024844] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.59@tcp (stopping) [ 932.027237] Lustre: Skipped 1 previous similar message [ 935.382848] Lustre: server umount lustre-MDT0000 complete [ 935.904470] LustreError: 44889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.910648] LustreError: 44889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 936.697190] LustreError: 44869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882225 with bad export cookie 12806725414174130309 [ 936.700270] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 936.701058] LustreError: 44869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 937.138265] LustreError: 44885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.149644] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.59@tcp (stopping) [ 941.023857] LustreError: 44884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.024303] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.257062] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.59@tcp (stopping) [ 942.259821] Lustre: Skipped 1 previous similar message [ 942.889899] Lustre: server umount lustre-MDT0001 complete [ 969.183099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 969.239852] Lustre: server umount lustre-OST0000 complete [ 974.799392] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 975.860339] Key type lgssc unregistered [ 975.989394] LNet: 48769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.992570] LNetError: 48769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.001836] LNet: Removed LNI 192.168.201.159@tcp [ 976.307735] Key type .llcrypt unregistered [ 976.309090] Key type ._llcrypt unregistered [ 986.299812] Key type ._llcrypt registered [ 986.301183] Key type .llcrypt registered [ 986.348389] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 986.750350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.765347] alg: No test for adler32 (adler32-zlib) [ 987.620324] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 987.725085] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 989.311156] Key type lgssc registered [ 989.684968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 992.370575] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 992.377357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.839303] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1012.959296] LustreError: 50145:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 1039.583153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.873670] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1044.576328] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1209.852314] Lustre: server umount lustre-OST0000 complete [ 1214.731649] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1215.644717] Key type lgssc unregistered [ 1215.765359] LNet: 51818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.767323] LNetError: 51818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.773290] LNet: Removed LNI 192.168.201.159@tcp [ 1216.043736] Key type .llcrypt unregistered [ 1216.045112] Key type ._llcrypt unregistered [ 1223.799394] Key type ._llcrypt registered [ 1223.800343] Key type .llcrypt registered [ 1223.831304] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1224.108822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1224.115352] alg: No test for adler32 (adler32-zlib) [ 1224.953054] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1225.031516] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1226.607142] Key type lgssc registered [ 1226.910799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.233255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1229.238026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.363124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.559805] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1234.105866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.213127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.417401] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1237.702579] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1239.336329] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1241.407507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1241.519979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.413855] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1245.801544] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1248.738058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1248.740428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.744524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1252.683073] Lustre: server umount lustre-OST0000 complete [ 1254.879803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1254.882428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.887481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.976226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.976715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.096119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.097880] Lustre: Skipped 2 previous similar messages [ 1268.191157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1268.249531] Lustre: server umount lustre-MDT0000 complete [ 1269.216309] LustreError: 53202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.221210] LustreError: 53202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1269.502222] LustreError: 55041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882558 with bad export cookie 9784515319871435173 [ 1269.505348] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1269.505396] LustreError: 55041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1269.638177] Lustre: server umount lustre-MDT0001 complete [ 1273.427755] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1274.370851] Key type lgssc unregistered [ 1274.497334] LNet: 56733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.499844] LNetError: 56733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.507258] LNet: Removed LNI 192.168.201.159@tcp [ 1274.790082] Key type .llcrypt unregistered [ 1274.791192] Key type ._llcrypt unregistered [ 1283.273626] Key type ._llcrypt registered [ 1283.274591] Key type .llcrypt registered [ 1283.317531] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1283.742097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1283.784195] alg: No test for adler32 (adler32-zlib) [ 1284.639186] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1284.727227] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1286.303119] Key type lgssc registered [ 1286.631625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1289.116841] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1289.122348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1304.543355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1309.663308] LustreError: 58092:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 1336.287210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.222955] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1340.566621] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1342.763848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.937727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.962785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1344.290198] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1347.086655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.222831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1348.612814] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1351.055944] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1352.854739] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1356.771240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.772079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1356.777971] Lustre: Skipped 1 previous similar message [ 1359.628898] Lustre: server umount lustre-OST0000 complete [ 1373.715990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.720397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.723624] Lustre: Skipped 1 previous similar message [ 1376.801070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.801090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1376.807413] Lustre: Skipped 1 previous similar message [ 1379.993145] Lustre: server umount lustre-MDT0000 complete [ 1381.236265] LustreError: 60701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882669 with bad export cookie 7580612355203152784 [ 1381.239473] LustreError: 60701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1381.240317] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1387.433792] Lustre: server umount lustre-MDT0001 complete [ 1390.842897] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1391.811165] Key type lgssc unregistered [ 1391.935322] LNet: 61833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.937550] LNetError: 61833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.947306] LNet: Removed LNI 192.168.201.159@tcp [ 1392.257778] Key type .llcrypt unregistered [ 1392.258880] Key type ._llcrypt unregistered [ 1400.907597] Key type ._llcrypt registered [ 1400.909089] Key type .llcrypt registered [ 1400.956677] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1401.266732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.291066] alg: No test for adler32 (adler32-zlib) [ 1402.140350] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1402.234820] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1403.815133] Key type lgssc registered [ 1404.181868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1406.778386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1406.784947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.929827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.210800] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1411.952362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.072754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1413.460896] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1415.883443] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1417.732216] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1419.995174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.120140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.214715] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1424.698500] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1432.422544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1437.665322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1437.665960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.668675] LustreError: Skipped 1 previous similar message [ 1437.669547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.673856] Lustre: Skipped 1 previous similar message [ 1437.679147] Lustre: Skipped 1 previous similar message [ 1439.247718] Lustre: server umount lustre-OST0000 complete [ 1442.586806] Lustre: server umount lustre-MDT0000 complete [ 1442.783874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1442.786159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.791060] LustreError: 63216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.795475] LustreError: 63216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1443.834131] LustreError: 63199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882732 with bad export cookie 13597543329976070729 [ 1443.837704] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1443.838243] LustreError: 63199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1444.017633] Lustre: server umount lustre-MDT0001 complete [ 1447.615358] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1448.710024] Key type lgssc unregistered [ 1448.848317] LNet: 66765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.851270] LNetError: 66765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.858380] LNet: Removed LNI 192.168.201.159@tcp [ 1449.180907] Key type .llcrypt unregistered [ 1449.182710] Key type ._llcrypt unregistered [ 1458.841972] Key type ._llcrypt registered [ 1458.843009] Key type .llcrypt registered [ 1458.881822] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1459.218330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1459.226961] alg: No test for adler32 (adler32-zlib) [ 1460.079313] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1460.180304] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1461.759195] Key type lgssc registered [ 1462.105613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1465.152362] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 11:52:33 (1788882753) [ 1467.344800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1467.351480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.507918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.849362] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1472.789185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.912382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1474.295306] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1476.810764] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1478.701907] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1481.003641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1481.133423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.110427] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1485.597953] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1493.414734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1498.391387] Lustre: Failing over lustre-MDT0000 [ 1498.565452] Lustre: server umount lustre-MDT0000 complete [ 1498.594964] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.597444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1498.597571] LustreError: 68437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.597580] LustreError: 68437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1498.600130] Lustre: Skipped 1 previous similar message [ 1501.367836] LustreError: 70929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.378107] LustreError: 70929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1501.946727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.994864] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1502.064229] Lustre: *** cfs_fail_loc=135, val=0*** [ 1502.118527] LustreError: 71548:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1502.120991] LustreError: 71548:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.159@tcp: cfg command failed: rc = -2 [ 1502.124048] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1502.124048] [ 1502.127759] LustreError: MGC192.168.201.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1502.132700] LustreError: 71536:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1502.136175] LustreError: 71536:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 1502.140118] LustreError: 71536:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1502.142774] LustreError: 71536:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1502.156079] Lustre: server umount lustre-MDT0000 complete [ 1502.157632] LustreError: 71536:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1503.712841] LustreError: 68441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.718167] LustreError: 68441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1504.471410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.522205] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1504.624634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.648816] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1506.061209] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1506.481430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1509.865078] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1509.876602] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1509.894674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1513.952079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.955594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.961142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1514.977929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1514.978307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1514.981124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1516.940987] Lustre: server umount lustre-OST0000 complete [ 1520.096394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.096832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.104651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.106720] Lustre: Skipped 2 previous similar messages [ 1524.379173] Lustre: server umount lustre-MDT0000 complete [ 1525.216082] LustreError: 68441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.221155] LustreError: 68441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1525.636224] LustreError: 68422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882814 with bad export cookie 13136026402627973595 [ 1525.639517] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1525.639765] LustreError: 68422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1525.760816] Lustre: server umount lustre-MDT0001 complete [ 1529.313729] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1530.430488] Key type lgssc unregistered [ 1530.571504] LNet: 73323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.575210] LNetError: 73323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.582374] LNet: Removed LNI 192.168.201.159@tcp [ 1530.936932] Key type .llcrypt unregistered [ 1530.938032] Key type ._llcrypt unregistered [ 1540.374661] Key type ._llcrypt registered [ 1540.375626] Key type .llcrypt registered [ 1540.415460] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1540.734031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.743453] alg: No test for adler32 (adler32-zlib) [ 1541.599684] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1541.695410] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1543.279154] Key type lgssc registered [ 1543.637392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1546.184491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1546.190483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.336565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.692703] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1551.437695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.564081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1552.889289] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1555.254188] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1556.973334] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1559.218192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1559.341594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.353509] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1563.707309] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1572.645902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1576.190066] Lustre: Failing over lustre-MDT0000 [ 1576.342619] Lustre: server umount lustre-MDT0000 complete [ 1577.440935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1577.444219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.448181] LustreError: 74707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.454227] LustreError: 74707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1577.953989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.954986] LustreError: 74702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.958383] Lustre: Skipped 1 previous similar message [ 1579.190586] LustreError: 74703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.194954] LustreError: 74703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1579.320218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.359450] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.413422] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1579.470401] LustreError: 77814:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1579.472810] LustreError: 77814:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.159@tcp: cfg command failed: rc = -9 [ 1579.475941] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1579.475941] [ 1579.479724] LustreError: MGC192.168.201.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1579.484607] LustreError: 77802:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1579.487196] LustreError: 77802:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 1579.490267] LustreError: 77802:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1579.492017] LustreError: 77802:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1579.502842] Lustre: server umount lustre-MDT0000 complete [ 1579.504623] LustreError: 77802:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1581.706668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.747794] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1581.845767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.877556] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1583.343828] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1584.305565] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1587.176442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1587.189862] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1587.210258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1592.291865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.291984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1592.295943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1592.297520] LustreError: Skipped 1 previous similar message [ 1592.300787] Lustre: Skipped 1 previous similar message [ 1594.191365] Lustre: server umount lustre-OST0000 complete [ 1597.409093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1597.409194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1597.409804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1597.412843] Lustre: Skipped 2 previous similar messages [ 1601.557383] Lustre: server umount lustre-MDT0000 complete [ 1602.529106] LustreError: 74707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.537022] LustreError: 74707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1602.766183] LustreError: 74689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882891 with bad export cookie 17206800097683260929 [ 1602.768947] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1602.771023] LustreError: 74689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1602.891660] Lustre: server umount lustre-MDT0001 complete [ 1606.317159] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1607.324409] Key type lgssc unregistered [ 1607.455316] LNet: 79589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.458057] LNetError: 79589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.463259] LNet: Removed LNI 192.168.201.159@tcp [ 1607.771496] Key type .llcrypt unregistered [ 1607.772776] Key type ._llcrypt unregistered [ 1611.846907] Key type ._llcrypt registered [ 1611.847842] Key type .llcrypt registered [ 1611.894583] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1612.009367] Key type .llcrypt unregistered [ 1612.010400] Key type ._llcrypt unregistered [ 1620.635268] Key type ._llcrypt registered [ 1620.636920] Key type .llcrypt registered [ 1620.678609] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1621.095802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1621.153927] alg: No test for adler32 (adler32-zlib) [ 1622.013705] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1622.104462] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1623.679157] Key type lgssc registered [ 1624.091354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1626.801816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1626.809856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.993309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.480311] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1632.469914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.621649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1634.036398] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1636.525186] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1638.320298] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1640.544398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1640.686165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.600954] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1645.143942] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1652.967224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1658.339286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1658.339816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.349039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1658.350889] Lustre: Skipped 1 previous similar message [ 1663.461441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.463168] Lustre: Skipped 1 previous similar message [ 1664.212111] Lustre: server umount lustre-OST0000 complete [ 1668.578076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.578100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1668.581827] Lustre: Skipped 1 previous similar message [ 1668.583995] LustreError: Skipped 1 previous similar message [ 1668.585776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.588079] Lustre: Skipped 1 previous similar message [ 1671.641560] Lustre: server umount lustre-MDT0000 complete [ 1672.858179] LustreError: 83955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882961 with bad export cookie 14641577120756067540 [ 1672.861103] LustreError: 83955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1672.861894] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.982532] Lustre: server umount lustre-MDT0001 complete [ 1676.177555] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1677.134928] Key type lgssc unregistered [ 1677.265362] LNet: 85009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.267369] LNetError: 85009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.276306] LNet: Removed LNI 192.168.201.159@tcp [ 1677.575512] Key type .llcrypt unregistered [ 1677.576513] Key type ._llcrypt unregistered [ 1685.897978] Key type ._llcrypt registered [ 1685.899024] Key type .llcrypt registered [ 1685.937383] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1686.257178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1686.283763] alg: No test for adler32 (adler32-zlib) [ 1687.137687] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1687.237281] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1688.815179] Key type lgssc registered [ 1689.137182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.591142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1691.596855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.747771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.149087] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1696.965804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.101080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1698.421745] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1700.767765] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1702.564589] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1704.951580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1705.086090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.049690] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1709.425644] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1718.443336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1722.848872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1722.852778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.858900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1723.874134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1723.874506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1723.878826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.234604] Lustre: server umount lustre-OST0000 complete [ 1727.506633] Lustre: server umount lustre-MDT0000 complete [ 1727.967799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1727.970166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.975244] LustreError: 86394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.982442] LustreError: 86394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1728.678419] LustreError: 86376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883017 with bad export cookie 10232888250356560653 [ 1728.681429] LustreError: 86376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1728.681964] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.798404] Lustre: server umount lustre-MDT0001 complete [ 1732.191817] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1733.189607] Key type lgssc unregistered [ 1733.296401] LNet: 89939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.298465] LNetError: 89939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.305251] LNet: Removed LNI 192.168.201.159@tcp [ 1733.606221] Key type .llcrypt unregistered [ 1733.607609] Key type ._llcrypt unregistered [ 1742.079236] Key type ._llcrypt registered [ 1742.080093] Key type .llcrypt registered [ 1742.118160] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1742.448406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.488050] alg: No test for adler32 (adler32-zlib) [ 1743.337848] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1743.425079] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1744.999217] Key type lgssc registered [ 1745.340903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1747.809200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1747.817364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.966381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.232061] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1752.883138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.999678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1754.266768] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1756.582145] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1758.291747] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1760.373461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.498586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.297534] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1764.461455] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1773.794509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1778.143823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1778.146940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.151971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1779.169382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1779.170288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.170921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.817801] Lustre: server umount lustre-OST0000 complete [ 1788.895804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1788.897722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.901100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.902561] Lustre: Skipped 1 previous similar message [ 1791.189868] Lustre: server umount lustre-MDT0000 complete [ 1792.344135] LustreError: 91303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883081 with bad export cookie 6190099326639766888 [ 1792.347108] LustreError: 91303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1792.347574] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.453235] Lustre: server umount lustre-MDT0001 complete [ 1795.915391] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1796.905955] Key type lgssc unregistered [ 1797.033355] LNet: 94872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.035355] LNetError: 94872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.042267] LNet: Removed LNI 192.168.201.159@tcp [ 1797.418712] Key type .llcrypt unregistered [ 1797.420593] Key type ._llcrypt unregistered [ 1806.404882] Key type ._llcrypt registered [ 1806.406158] Key type .llcrypt registered [ 1806.445516] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1806.745213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.752840] alg: No test for adler32 (adler32-zlib) [ 1807.623841] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1807.717888] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1809.303225] Key type lgssc registered [ 1809.643000] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.561305] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1812.567837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1827.999479] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1833.119347] LustreError: 96231:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 1859.743376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.096974] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1865.069618] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1867.706250] Lustre: server umount lustre-OST0000 complete [ 1871.389817] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 11:59:19 (1788883159) [ 1874.406089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.583137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.299941] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1879.922134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.065497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1881.815906] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1885.146531] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.891158] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1889.305675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.450592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.742744] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1894.422745] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1900.719190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1911.268228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.269708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.273187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.274084] LustreError: Skipped 1 previous similar message [ 1911.286045] Lustre: Skipped 1 previous similar message [ 1914.919954] Lustre: server umount lustre-OST0000 complete [ 1920.992388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1920.997645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.006651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.010273] Lustre: Skipped 1 previous similar message [ 1922.670672] Lustre: server umount lustre-MDT0000 complete [ 1923.975980] LustreError: 100622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883212 with bad export cookie 359290341905987183 [ 1923.978199] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1923.981228] LustreError: 100622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1924.103616] Lustre: server umount lustre-MDT0001 complete [ 1928.042352] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1929.268694] Key type lgssc unregistered [ 1929.408507] LNet: 101572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.410930] LNetError: 101572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.417465] LNet: Removed LNI 192.168.201.159@tcp [ 1929.765392] Key type .llcrypt unregistered [ 1929.766407] Key type ._llcrypt unregistered [ 1945.825213] Key type ._llcrypt registered [ 1945.826232] Key type .llcrypt registered [ 1945.865648] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1946.187339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1946.195319] alg: No test for adler32 (adler32-zlib) [ 1947.047944] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1947.133778] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1948.719207] Key type lgssc registered [ 1949.199093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.994917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1952.005805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.205404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.749900] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1957.894832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.036413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1959.484166] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1962.705699] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1964.390357] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1967.251746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.404071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.267926] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1972.237220] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1977.638762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1988.065623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1988.066623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.071486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1988.072996] Lustre: Skipped 1 previous similar message [ 1991.243626] Lustre: server umount lustre-OST0000 complete [ 1993.184749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.185811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.189399] Lustre: Skipped 2 previous similar messages [ 1993.193725] Lustre: Skipped 1 previous similar message [ 1998.304018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.305863] Lustre: Skipped 1 previous similar message [ 1998.549766] Lustre: server umount lustre-MDT0000 complete [ 1999.729907] LustreError: 103709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883288 with bad export cookie 632872341648050362 [ 1999.732639] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1999.734675] LustreError: 103709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1999.841829] Lustre: server umount lustre-MDT0001 complete [ 2003.111388] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2004.034523] Key type lgssc unregistered [ 2004.154392] LNet: 107293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.156424] LNetError: 107293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.163290] LNet: Removed LNI 192.168.201.159@tcp [ 2004.463137] Key type .llcrypt unregistered [ 2004.464231] Key type ._llcrypt unregistered [ 2011.273450] Key type ._llcrypt registered [ 2011.274428] Key type .llcrypt registered [ 2011.308761] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2011.628655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.656253] alg: No test for adler32 (adler32-zlib) [ 2012.503227] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2012.586154] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2014.159136] Key type lgssc registered [ 2014.495819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2017.031979] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2017.039029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2032.479402] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2037.599374] LustreError: 108507:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 2064.223199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.045240] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2068.331257] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2070.376504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.467378] LustreError: MGC192.168.201.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2070.471516] LustreError: 109532:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2070.473952] LustreError: 109532:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2070.476588] LustreError: 109532:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2070.504818] Lustre: server umount lustre-MDT0000 complete [ 2070.506262] LustreError: 109532:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2083.334306] Lustre: server umount lustre-OST0000 complete [ 2086.130950] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 2088.851439] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2099.492552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.577955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2099.590301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2099.627177] Lustre: lustre-MDT0000: new disk, initializing [ 2099.647713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.653088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2101.073352] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2105.739553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.788654] Lustre: 112317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2105.816267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2105.818294] Lustre: Skipped 1 previous similar message [ 2105.855743] Lustre: lustre-MDT0001: new disk, initializing [ 2105.894325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2105.909175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2105.912971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2107.678264] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2112.710260] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2115.355440] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2118.602528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2118.765320] Lustre: lustre-OST0000: new disk, initializing [ 2118.768462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2118.772639] Lustre: 114286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2118.828792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.419098] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2123.848603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2123.862026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2123.957806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2130.905489] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2134.233182] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2134.388349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2137.451496] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2137.592615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2139.625550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2139.627161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.646549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2139.649850] Lustre: Skipped 1 previous similar message [ 2139.662060] Lustre: Skipped 1 previous similar message [ 2144.742147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2144.745823] Lustre: Skipped 1 previous similar message [ 2144.921620] Lustre: server umount lustre-OST0000 complete [ 2146.785522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2146.788730] LustreError: Skipped 1 previous similar message [ 2146.790747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.797201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.856686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.857395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.483452] Lustre: server umount lustre-MDT0000 complete [ 2153.995336] LustreError: 112311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883442 with bad export cookie 12890897412246243177 [ 2153.999148] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.999787] LustreError: 112311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2154.119928] Lustre: server umount lustre-MDT0001 complete [ 2157.002817] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 12:04:05 (1788883445) [ 2158.106287] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2163.626415] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 2166.896469] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2179.986468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.077063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2180.089586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2180.128933] Lustre: lustre-MDT0000: new disk, initializing [ 2180.162063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.169150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2181.772163] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2186.424588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.477808] Lustre: 119178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2186.505625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2186.508421] Lustre: Skipped 1 previous similar message [ 2186.551365] Lustre: lustre-MDT0001: new disk, initializing [ 2186.610897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2186.613943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2188.099671] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2192.099182] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2194.047888] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2196.471719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2196.573480] Lustre: lustre-OST0000: new disk, initializing [ 2196.574863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2196.577347] Lustre: 121142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2196.605441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.607913] Lustre: Skipped 1 previous similar message [ 2198.550783] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2201.086642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2201.091060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2201.137158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2202.423048] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2203.949677] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2204.022303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2205.502778] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2205.575366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2211.297322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2211.298321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.299779] LustreError: Skipped 1 previous similar message [ 2211.300350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2211.303841] Lustre: Skipped 1 previous similar message [ 2211.304892] Lustre: Skipped 1 previous similar message [ 2212.369805] Lustre: server umount lustre-OST0000 complete [ 2216.416729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.422423] Lustre: Skipped 1 previous similar message [ 2219.679498] Lustre: server umount lustre-MDT0000 complete [ 2220.979199] LustreError: 119168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883509 with bad export cookie 12890897412246244059 [ 2220.982578] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2220.983482] LustreError: 119168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2221.103481] Lustre: server umount lustre-MDT0001 complete [ 2224.304826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.506704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.061524] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2229.171760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.818142] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2233.552762] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.501628] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.853643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2240.000108] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2242.536708] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2255.447671] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2260.447891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2260.450637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.456541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2260.459308] Lustre: Skipped 3 previous similar messages [ 2262.545210] Lustre: server umount lustre-OST0000 complete [ 2265.055911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2265.059272] LustreError: Skipped 1 previous similar message [ 2276.832165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.834850] Lustre: Skipped 8 previous similar messages [ 2278.367102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2278.425470] Lustre: server umount lustre-MDT0000 complete [ 2279.769290] LustreError: 124090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883568 with bad export cookie 12890897412246244941 [ 2279.772329] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.776154] LustreError: 124090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2279.888370] Lustre: server umount lustre-MDT0001 complete [ 2283.758032] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2284.991740] Key type lgssc unregistered [ 2285.143396] LNet: 127006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.148200] LNetError: 127006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.158404] LNet: Removed LNI 192.168.201.159@tcp [ 2285.501338] Key type .llcrypt unregistered [ 2285.502782] Key type ._llcrypt unregistered [ 2292.281132] Key type ._llcrypt registered [ 2292.282756] Key type .llcrypt registered [ 2292.329195] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 2298.958709] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2299.433126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.446878] alg: No test for adler32 (adler32-zlib) [ 2300.337799] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2300.448810] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2302.040723] Key type lgssc registered [ 2302.458619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.922871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.929826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.016278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2316.029338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2316.068277] Lustre: lustre-MDT0000: new disk, initializing [ 2316.103385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2316.111274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2317.695185] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2322.335560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.382874] Lustre: 130127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2322.405371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2322.407407] Lustre: Skipped 1 previous similar message [ 2322.444413] Lustre: lustre-MDT0001: new disk, initializing [ 2322.478699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2322.492618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2322.495835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2323.988772] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2328.016212] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.948403] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2332.349446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.458631] Lustre: lustre-OST0000: new disk, initializing [ 2332.460390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2332.464267] Lustre: 132096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2332.496435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.737197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2332.741460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2332.778479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2334.511252] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2338.417328] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2339.963220] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2340.047728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2341.545733] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2341.632332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2342.880491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2342.885356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.891634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2343.392735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2343.393653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2343.395287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.500791] Lustre: server umount lustre-OST0000 complete [ 2353.120061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2353.122783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.129125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.131523] Lustre: Skipped 1 previous similar message [ 2358.752673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.755396] Lustre: Skipped 2 previous similar messages [ 2363.871968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2363.874871] Lustre: Skipped 1 previous similar message [ 2364.383123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2364.454339] Lustre: server umount lustre-MDT0000 complete [ 2365.850446] LustreError: 130120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883654 with bad export cookie 7106043677091306345 [ 2365.853379] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.856138] LustreError: 130120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2365.984586] Lustre: server umount lustre-MDT0001 complete [ 2368.861831] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 12:07:37 (1788883657) [ 2371.075583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.249598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.588333] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2375.631450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.764944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.316404] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2380.039511] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2381.863940] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2385.632875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.633746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2385.637128] Lustre: Skipped 2 previous similar messages [ 2388.689208] Lustre: server umount lustre-MDT0000 complete [ 2389.907276] LustreError: 134485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883678 with bad export cookie 7106043677091307227 [ 2389.910383] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.911648] LustreError: 134485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2390.024183] Lustre: server umount lustre-MDT0001 complete [ 2392.701548] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 12:08:01 (1788883681) [ 2394.818661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2410.271574] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2415.391334] LustreError: 136915:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 2440.991445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.991890] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2445.426779] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2446.156103] Lustre: server umount lustre-OST0000 complete [ 2448.858092] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 12:08:57 (1788883737) [ 2450.805958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.935557] LustreError: 138358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.941615] LustreError: 138358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2452.161533] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2454.708024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.092242] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2458.408489] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2460.113196] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2462.316928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2462.421968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.423986] Lustre: Skipped 2 previous similar messages [ 2464.197155] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2466.461887] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2480.607822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2480.610072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.613638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.615201] Lustre: Skipped 1 previous similar message [ 2484.307744] Lustre: server umount lustre-MDT0000 complete [ 2485.485990] LustreError: 138338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883774 with bad export cookie 7106043677091307871 [ 2485.488967] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.489959] LustreError: 138338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2485.592670] Lustre: server umount lustre-MDT0001 complete [ 2496.991508] Lustre: server umount lustre-OST0000 complete [ 2499.540619] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 12:09:48 (1788883788) [ 2501.613892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.757406] LustreError: 142030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.780103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.076399] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2505.826096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.273931] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2509.550116] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2511.229098] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2513.365492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2515.342706] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2517.688741] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2518.970292] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2527.845649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2528.198857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2529.494232] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2529.561199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2531.295756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2531.298586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.303119] Lustre: Skipped 2 previous similar messages [ 2531.305341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.307142] Lustre: Skipped 3 previous similar messages [ 2536.266142] Lustre: server umount lustre-OST0000 complete [ 2543.510639] Lustre: server umount lustre-MDT0000 complete [ 2543.583893] LustreError: 142030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.589803] LustreError: 142030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2544.711921] LustreError: 142011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883833 with bad export cookie 7106043677091308928 [ 2544.714449] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2544.715947] LustreError: 142011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2544.818038] Lustre: server umount lustre-MDT0001 complete [ 2547.403513] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 12:10:35 (1788883835) [ 2549.521735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2564.959539] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2570.079346] LustreError: 146256:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 2595.679332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.680883] Lustre: Skipped 2 previous similar messages [ 2597.450613] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2599.697735] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2601.793961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.927490] LustreError: 146281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.971407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2603.194123] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2605.823128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.214541] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2609.461528] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2611.203675] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2612.527846] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2612.599253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2613.867811] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2613.939511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2616.288752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2616.289729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.290766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.290770] Lustre: Skipped 4 previous similar messages [ 2616.291248] LustreError: Skipped 1 previous similar message [ 2616.299018] Lustre: Skipped 5 previous similar messages [ 2620.694317] Lustre: server umount lustre-MDT0000 complete [ 2621.408856] LustreError: 147292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.413602] LustreError: 147292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2621.914969] LustreError: 146263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883910 with bad export cookie 7106043677091309943 [ 2621.918016] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.919596] LustreError: 146263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2622.025668] Lustre: server umount lustre-MDT0001 complete [ 2632.408437] Lustre: server umount lustre-OST0000 complete [ 2635.019855] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 12:12:03 (1788883923) [ 2637.217011] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.639537] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2657.759429] LustreError: 150445:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 2685.188068] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2687.463413] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2689.567178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.702059] LustreError: 150470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.744953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2690.936083] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2693.443261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.782913] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2697.033706] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2698.765698] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2700.882471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2700.951445] Lustre: lustre-OST0001: new disk, initializing [ 2700.953330] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2700.955709] Lustre: 153314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2702.765863] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2703.350955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2703.354475] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2703.381199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2706.295290] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2707.611939] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2707.679488] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2708.978871] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2709.046387] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2710.495733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2710.498176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.702291] Lustre: server umount lustre-OST0000 complete [ 2718.688455] LustreError: 153774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.693043] LustreError: 153774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2722.962535] Lustre: server umount lustre-OST0001 complete [ 2730.322813] Lustre: server umount lustre-MDT0000 complete [ 2731.456236] LustreError: 151507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884020 with bad export cookie 7106043677091310895 [ 2731.459860] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2731.460252] LustreError: 151507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2731.565045] Lustre: server umount lustre-MDT0001 complete [ 2738.487625] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 12:13:47 (1788884027) [ 2738.950077] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2739.384180] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2739.846898] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 12:13:48 (1788884028) [ 2742.067792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.161653] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2742.171479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2742.246437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.248445] Lustre: Skipped 6 previous similar messages [ 2743.618401] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2746.486469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.520725] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2746.531580] Lustre: 156869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2748.045347] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2750.514091] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2752.233076] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2754.366791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.430879] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2756.314671] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2756.939836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2758.578234] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2759.803726] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2759.866692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2761.037399] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2761.097361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2762.208831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.210858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.211126] LustreError: Skipped 4 previous similar messages [ 2762.212516] Lustre: Skipped 10 previous similar messages [ 2763.729142] Lustre: server umount lustre-OST0000 complete [ 2777.585269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.697968] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2782.077733] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2782.823110] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.826902] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2782.826981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.679940] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2783.759706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2785.200445] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2785.289762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2787.028429] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e82c3711000.ost_server_uuid 50 [ 2787.488717] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e82c3711000.ost_server_uuid in FULL state after 0 sec [ 2787.542525] LustreError: 156877:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.59@tcp) returned error from blocking AST (req@00000000ace898d8 x1875780398406400 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000726fa986/0x629dbd132b70367b lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.59@tcp remote: 0x7b0d381e95084f97 expref: 5 pid: 156877 timeout: 2887 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2787.552900] LustreError: lustre-MDT0000: A client on nid 192.168.201.59@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2787.555592] LustreError: 156867:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.59@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000726fa986/0x629dbd132b70367b lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.59@tcp remote: 0x7b0d381e95084f97 expref: 6 pid: 156877 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2792.929336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.932326] Lustre: Skipped 7 previous similar messages [ 2794.634440] Lustre: server umount lustre-OST0000 complete [ 2810.335106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2810.401363] Lustre: server umount lustre-MDT0000 complete [ 2811.618249] LustreError: 156907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884100 with bad export cookie 7106043677091312078 [ 2811.621909] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.622468] LustreError: 156907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2817.773729] Lustre: server umount lustre-MDT0001 complete [ 2821.042239] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2822.131865] Key type lgssc unregistered [ 2822.277376] LNet: 162973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2822.279825] LNetError: 162973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2822.286315] LNet: Removed LNI 192.168.201.159@tcp [ 2822.600070] Key type .llcrypt unregistered [ 2822.601251] Key type ._llcrypt unregistered [ 2831.713112] Key type ._llcrypt registered [ 2831.714749] Key type .llcrypt registered [ 2831.758232] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2832.139261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2832.212890] alg: No test for adler32 (adler32-zlib) [ 2833.077664] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2833.177121] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2834.759090] Key type lgssc registered [ 2835.136783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2837.836261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2837.842193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.992278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.331173] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2843.191502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.311066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2844.704849] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2847.182815] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2849.107377] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2851.373987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.486845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.362903] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2855.714377] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2863.783496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2868.526973] Lustre: Failing over lustre-MDT0000 [ 2868.724397] Lustre: server umount lustre-MDT0000 complete [ 2869.217439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2869.218673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.220914] LustreError: 164356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.224241] Lustre: Skipped 2 previous similar messages [ 2869.229464] LustreError: 164356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2874.337422] LustreError: 164375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.342996] LustreError: 164375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2879.456374] LustreError: 164357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.461032] LustreError: 164357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2879.605839] LustreError: 164383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.59@tcp arrived at 1788884168 with bad export cookie 1610267281098516150 [ 2879.611381] LustreError: 164383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2882.009895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.049268] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2882.142163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.165656] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2883.489205] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2884.188063] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2884.190528] Lustre: lustre-MDT0000: Denying connection for new client f1670132-bebe-430a-8757-98dde8476213 (at 192.168.201.59@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2887.652947] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2892.770728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.772523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2892.776849] Lustre: Skipped 1 previous similar message [ 2892.779426] Lustre: Skipped 1 previous similar message [ 2897.042091] Lustre: server umount lustre-OST0000 complete [ 2898.237995] LustreError: 168101:0:(ldlm_lib.c:3031:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2898.240458] Lustre: 167452:0:(ldlm_lib.c:2431:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2898.243620] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2898.249456] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2898.253496] LustreError: 167452:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b546a13d x1875780956097024/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2898.260867] LustreError: 167452:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2898.265295] LustreError: 167452:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2898.270507] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2898.370186] Lustre: server umount lustre-MDT0000 complete [ 2899.582225] LustreError: 164337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884188 with bad export cookie 1610267281098516479 [ 2899.586230] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2899.586707] LustreError: 163667:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2905.787511] Lustre: server umount lustre-MDT0001 complete [ 2909.199234] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2910.239446] Key type lgssc unregistered [ 2910.364322] LNet: 168853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2910.366803] LNetError: 168853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2910.374279] LNet: Removed LNI 192.168.201.159@tcp [ 2910.687554] Key type .llcrypt unregistered [ 2910.688520] Key type ._llcrypt unregistered [ 2919.170461] Key type ._llcrypt registered [ 2919.171527] Key type .llcrypt registered [ 2919.208946] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2919.529824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2919.547071] alg: No test for adler32 (adler32-zlib) [ 2920.396335] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2920.495239] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2922.079173] Key type lgssc registered [ 2922.451099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2925.019700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2925.027233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.174914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.584867] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2930.419515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2930.549761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2931.973598] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2934.330322] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2936.142713] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2938.493150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2938.618815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.623234] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2943.141527] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2950.886367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2956.257285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2956.260282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.261205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.264708] Lustre: Skipped 1 previous similar message [ 2957.200721] Lustre: server umount lustre-OST0000 complete [ 2960.533351] Lustre: server umount lustre-MDT0000 complete [ 2961.376653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2961.377030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.380356] LustreError: Skipped 1 previous similar message [ 2961.389267] LustreError: 170240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.394719] LustreError: 170240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2961.798756] LustreError: 170221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884250 with bad export cookie 3494910232880030530 [ 2961.802530] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.803366] LustreError: 170221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2961.914232] Lustre: server umount lustre-MDT0001 complete [ 2965.428503] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2966.457750] Key type lgssc unregistered [ 2966.588434] LNet: 173787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2966.591136] LNetError: 173787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2966.597264] LNet: Removed LNI 192.168.201.159@tcp [ 2966.894750] Key type .llcrypt unregistered [ 2966.895889] Key type ._llcrypt unregistered [ 2977.861250] Key type ._llcrypt registered [ 2977.862254] Key type .llcrypt registered [ 2977.895211] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2978.162109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2978.168242] alg: No test for adler32 (adler32-zlib) [ 2979.014067] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2979.089709] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2980.663161] Key type lgssc registered [ 2981.046905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2983.635965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2983.641906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.781683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.123772] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2989.025112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.152754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.507717] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2993.023787] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2994.724936] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2996.743306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2996.849455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.589297] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3000.768549] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3011.173221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3014.624190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3014.628535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.635831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3016.673291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3016.674059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.674607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.504144] Lustre: server umount lustre-OST0000 complete [ 3019.743770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3019.746278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.750300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.751758] Lustre: Skipped 1 previous similar message [ 3021.792078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.792663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.758095] Lustre: server umount lustre-MDT0000 complete [ 3026.911953] LustreError: 175460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.916794] LustreError: 175460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3026.992175] LustreError: 175441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884315 with bad export cookie 8094140338759822007 [ 3026.995596] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.996928] LustreError: 175441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3027.107812] Lustre: server umount lustre-MDT0001 complete [ 3030.581531] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3031.684404] Key type lgssc unregistered [ 3031.817425] LNet: 179008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3031.821193] LNetError: 179008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3031.830362] LNet: Removed LNI 192.168.201.159@tcp [ 3032.147965] Key type .llcrypt unregistered [ 3032.149639] Key type ._llcrypt unregistered [ 3040.721489] Key type ._llcrypt registered [ 3040.722900] Key type .llcrypt registered [ 3040.764542] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3041.217395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3041.229811] alg: No test for adler32 (adler32-zlib) [ 3042.072106] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3042.152652] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3043.727127] Key type lgssc registered [ 3044.062760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3047.406206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3047.412039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.527852] Lustre: *** cfs_fail_loc=135, val=0*** [ 3048.564156] LustreError: 180504:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3048.566311] LustreError: 180504:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.159@tcp: cfg command failed: rc = -2 [ 3048.568925] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3048.568925] [ 3048.571172] LustreError: MGC192.168.201.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3048.574838] LustreError: 180466:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3048.577204] LustreError: 180466:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3048.579221] LustreError: 180466:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3048.580698] LustreError: 180466:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3048.608526] Lustre: server umount lustre-MDT0000 complete [ 3048.609815] LustreError: 180466:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3050.662478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.791264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.009896] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3054.555737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.658642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3055.864361] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3058.008598] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3059.619324] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3064.352655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.353391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.355620] Lustre: Skipped 1 previous similar message [ 3066.391193] Lustre: server umount lustre-MDT0000 complete [ 3067.496167] LustreError: 180855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884356 with bad export cookie 8962559975819435384 [ 3067.498954] LustreError: 180855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3067.498968] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.602108] Lustre: server umount lustre-MDT0001 complete [ 3070.748068] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3071.645990] Key type lgssc unregistered [ 3071.769295] LNet: 183190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3071.770960] LNetError: 183190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3071.778275] LNet: Removed LNI 192.168.201.159@tcp [ 3072.040164] Key type .llcrypt unregistered [ 3072.041074] Key type ._llcrypt unregistered [ 3078.052788] Key type ._llcrypt registered [ 3078.053697] Key type .llcrypt registered [ 3078.085624] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3078.171293] Key type .llcrypt unregistered [ 3078.172292] Key type ._llcrypt unregistered [ 3084.849523] Key type ._llcrypt registered [ 3084.851153] Key type .llcrypt registered [ 3084.889990] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3085.343684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3085.353982] alg: No test for adler32 (adler32-zlib) [ 3086.214721] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3086.312304] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3087.895198] Key type lgssc registered [ 3088.264416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3090.995986] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3091.002841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3106.463314] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3111.583381] LustreError: 185288:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 3138.207176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.139267] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3142.517734] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3144.760141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3144.928529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.947555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3146.255087] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3149.044689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.185166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3150.642733] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3153.187820] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3155.055571] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3156.013476] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3160.543882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3160.546083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.549724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3162.593589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3162.594434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.708861] Lustre: server umount lustre-OST0000 complete [ 3167.713765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.714460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3167.717536] Lustre: Skipped 1 previous similar message [ 3167.720292] Lustre: Skipped 1 previous similar message [ 3172.832868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3172.835554] Lustre: Skipped 2 previous similar messages [ 3172.950983] Lustre: server umount lustre-MDT0000 complete [ 3174.222343] LustreError: 187946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884462 with bad export cookie 5108185010061681683 [ 3174.225880] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3174.226579] LustreError: 187946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3174.345876] Lustre: server umount lustre-MDT0001 complete [ 3177.832713] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3178.773189] Key type lgssc unregistered [ 3178.891887] LNet: 189149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3178.893854] LNetError: 189149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3178.903302] LNet: Removed LNI 192.168.201.159@tcp [ 3179.211546] Key type .llcrypt unregistered [ 3179.213045] Key type ._llcrypt unregistered [ 3187.578192] Key type ._llcrypt registered [ 3187.579158] Key type .llcrypt registered [ 3187.615659] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3187.894969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3187.923122] alg: No test for adler32 (adler32-zlib) [ 3188.807742] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3188.911875] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3190.503167] Key type lgssc registered [ 3190.898486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3193.634095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3193.639169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.781178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.168354] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3199.074374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.202752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3200.534784] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3202.907371] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3204.773485] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3206.990589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3207.100628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.049394] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3211.440882] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3219.430493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3223.386826] Lustre: Failing over lustre-MDT0000 [ 3223.483145] Lustre: server umount lustre-MDT0000 complete [ 3224.545798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.546562] LustreError: 190529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.549120] Lustre: Skipped 2 previous similar messages [ 3224.554948] LustreError: 190529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3227.321958] LustreError: 190529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.665749] LustreError: 191964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.670209] LustreError: 191964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3232.432901] LustreError: 192738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.412730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.452341] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3236.542677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.577400] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3237.554215] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3237.931286] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3239.495599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3241.963451] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3241.972231] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3241.987736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3244.059350] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3250.143809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3250.145827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.146525] LustreError: Skipped 1 previous similar message [ 3250.147071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.153899] Lustre: Skipped 1 previous similar message [ 3252.192884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3252.195189] Lustre: Skipped 2 previous similar messages [ 3253.397963] Lustre: server umount lustre-OST0000 complete [ 3257.313457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3257.313772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.320591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3257.322547] Lustre: Skipped 1 previous similar message [ 3260.824704] Lustre: server umount lustre-MDT0000 complete [ 3262.138141] LustreError: 190516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884550 with bad export cookie 9370037433093968985 [ 3262.140852] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3262.142506] LustreError: 190516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3262.269852] Lustre: server umount lustre-MDT0001 complete [ 3266.058139] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3267.098435] Key type lgssc unregistered [ 3267.232420] LNet: 195429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3267.236872] LNetError: 195429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3267.247472] LNet: Removed LNI 192.168.201.159@tcp [ 3267.589853] Key type .llcrypt unregistered [ 3267.590995] Key type ._llcrypt unregistered [ 3276.682032] Key type ._llcrypt registered [ 3276.682996] Key type .llcrypt registered [ 3276.720617] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3277.067332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3277.114175] alg: No test for adler32 (adler32-zlib) [ 3277.997452] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3278.101779] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3279.687178] Key type lgssc registered [ 3280.076678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3282.670253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3282.676658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.820699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.138217] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3287.976183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.102954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3289.463607] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3291.909978] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3293.758934] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3296.028068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.141367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.033844] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3300.475071] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3308.455663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3311.431982] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3315.951965] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3327.619923] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3339.233584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3339.235186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.236380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3339.238695] Lustre: Skipped 1 previous similar message [ 3341.906955] Lustre: server umount lustre-OST0000 complete [ 3344.352723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3344.353383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.354259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3344.354264] Lustre: Skipped 1 previous similar message [ 3344.355165] LustreError: Skipped 1 previous similar message [ 3344.358615] Lustre: Skipped 1 previous similar message [ 3349.398032] Lustre: server umount lustre-MDT0000 complete [ 3349.472358] LustreError: 196812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.478236] LustreError: 196812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3350.743383] LustreError: 196793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884639 with bad export cookie 15545986724988171535 [ 3350.745907] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.748553] LustreError: 196793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3350.874043] Lustre: server umount lustre-MDT0001 complete [ 3354.596262] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3355.617242] Key type lgssc unregistered [ 3355.749348] LNet: 200673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3355.753094] LNetError: 200673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3355.761311] LNet: Removed LNI 192.168.201.159@tcp [ 3356.071740] Key type .llcrypt unregistered [ 3356.073228] Key type ._llcrypt unregistered [ 3363.562780] Key type ._llcrypt registered [ 3363.563889] Key type .llcrypt registered [ 3363.605781] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3364.030915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3364.043391] alg: No test for adler32 (adler32-zlib) [ 3364.894135] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3364.982368] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3366.559171] Key type lgssc registered [ 3366.982814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3369.759819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3369.767466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.942784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.354311] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3375.256384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.385450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3376.779138] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3379.227698] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3381.189266] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3383.529029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.651313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.599809] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3388.079667] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3395.941819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3400.243250] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3402.672259] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3411.680521] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3419.140061] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3426.785478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3426.787830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.789321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3426.790996] Lustre: Skipped 1 previous similar message [ 3431.508849] Lustre: server umount lustre-OST0000 complete [ 3437.025051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3437.025190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.025848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3437.025852] Lustre: Skipped 1 previous similar message [ 3437.027558] LustreError: Skipped 1 previous similar message [ 3437.030977] Lustre: Skipped 1 previous similar message [ 3442.144081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.145893] Lustre: Skipped 2 previous similar messages [ 3447.263123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3447.264429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3447.270070] Lustre: Skipped 1 previous similar message [ 3447.330040] Lustre: server umount lustre-MDT0000 complete [ 3448.575155] LustreError: 201893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884737 with bad export cookie 3639131156844650621 [ 3448.577992] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3448.578233] LustreError: 201893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3448.695261] Lustre: server umount lustre-MDT0001 complete [ 3452.327470] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3453.390409] Key type lgssc unregistered [ 3453.515390] LNet: 207139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3453.518502] LNetError: 207139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3453.527327] LNet: Removed LNI 192.168.201.159@tcp [ 3453.874287] Key type .llcrypt unregistered [ 3453.875335] Key type ._llcrypt unregistered [ 3463.045168] Key type ._llcrypt registered [ 3463.046141] Key type .llcrypt registered [ 3463.085644] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3463.415497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3463.440088] alg: No test for adler32 (adler32-zlib) [ 3464.298600] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3464.384869] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3465.967126] Key type lgssc registered [ 3466.341072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3468.915799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3468.921326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.062294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.471852] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3474.279155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.406548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3475.738704] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3478.177901] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3480.032033] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3482.356091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.487738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.494115] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3486.935580] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3494.758543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3503.010456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.011447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.013699] Lustre: Skipped 1 previous similar message [ 3503.016645] Lustre: Skipped 1 previous similar message [ 3507.154239] Lustre: server umount lustre-OST0000 complete [ 3510.239897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3510.242780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.248126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.312151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.312595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.517583] Lustre: server umount lustre-MDT0000 complete [ 3515.775280] LustreError: 209257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884804 with bad export cookie 6207665854861225371 [ 3515.777231] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3515.779040] LustreError: 209257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3515.902126] Lustre: server umount lustre-MDT0001 complete [ 3519.566103] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3520.608125] Key type lgssc unregistered [ 3520.736405] LNet: 212223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3520.738699] LNetError: 212223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3520.744295] LNet: Removed LNI 192.168.201.159@tcp [ 3521.075891] Key type .llcrypt unregistered [ 3521.077351] Key type ._llcrypt unregistered [ 3528.448479] Key type ._llcrypt registered [ 3528.449645] Key type .llcrypt registered [ 3528.492694] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3528.865520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3528.915843] alg: No test for adler32 (adler32-zlib) [ 3529.765653] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3529.865688] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3531.447135] Key type lgssc registered [ 3531.824739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3534.454334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3534.460343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.601241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.939892] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3539.776863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.903131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3541.217524] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3543.525181] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3545.360924] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3547.562864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3547.689783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.578544] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3551.973322] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3560.997251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3566.050588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.051508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3566.053985] Lustre: Skipped 1 previous similar message [ 3566.057604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3568.012824] Lustre: server umount lustre-OST0000 complete [ 3570.655786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3570.657824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.661377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3570.662877] Lustre: Skipped 1 previous similar message [ 3571.284873] Lustre: server umount lustre-MDT0000 complete [ 3572.491116] LustreError: 214196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884861 with bad export cookie 5563740789128717458 [ 3572.494049] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3572.494194] LustreError: 214196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3572.604892] Lustre: server umount lustre-MDT0001 complete [ 3576.095608] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3577.096271] Key type lgssc unregistered [ 3577.225389] LNet: 217160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3577.227666] LNetError: 217160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3577.234254] LNet: Removed LNI 192.168.201.159@tcp [ 3577.540641] Key type .llcrypt unregistered [ 3577.542419] Key type ._llcrypt unregistered [ 3584.689781] Key type ._llcrypt registered [ 3584.690906] Key type .llcrypt registered [ 3584.728330] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3585.055225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3585.104025] alg: No test for adler32 (adler32-zlib) [ 3585.953828] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3586.043870] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3587.631142] Key type lgssc registered [ 3588.037948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3590.682267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3590.687671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.822642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.116906] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3595.873867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.997645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.314237] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3599.683384] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3601.426117] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3603.611072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.726690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.609901] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3607.969280] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3618.087471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3620.856057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.879912] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3620.957389] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3622.883801] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3625.179313] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3625.673111] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3636.253780] Lustre: Permanently deactivating lustre-OST0001 [ 3636.258995] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3636.420930] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3643.973305] Lustre: Permanently reactivating lustre-OST0001 [ 3643.979556] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3643.981516] Lustre: Skipped 2 previous similar messages [ 3645.305909] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.310989] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3645.313432] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3645.316645] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3645.329778] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3655.648180] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3655.648357] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.650197] LustreError: Skipped 1 previous similar message [ 3655.653559] Lustre: Skipped 1 previous similar message [ 3655.655985] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3655.658046] Lustre: Skipped 1 previous similar message [ 3657.189265] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3657.191046] Lustre: Skipped 1 previous similar message [ 3661.435089] Lustre: server umount lustre-OST0001 complete [ 3662.309226] LustreError: 220609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3662.315932] LustreError: 220609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3664.351801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3664.354102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.357962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.426605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.426654] LustreError: 220218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.426823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3667.426868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.429573] Lustre: Skipped 1 previous similar message [ 3667.436592] LustreError: 220218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3668.757841] Lustre: server umount lustre-OST0000 complete [ 3672.545246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.552368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3672.555016] Lustre: Skipped 1 previous similar message [ 3682.784332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3682.787148] Lustre: Skipped 3 previous similar messages [ 3684.319236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3684.384966] Lustre: server umount lustre-MDT0000 complete [ 3685.618615] LustreError: 222035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884974 with bad export cookie 3150558850853549397 [ 3685.623125] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.623332] LustreError: 222035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3685.758855] Lustre: server umount lustre-MDT0001 complete [ 3689.590751] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3690.739949] Key type lgssc unregistered [ 3690.868361] LNet: 223480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3690.870741] LNetError: 223480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3690.884314] LNet: Removed LNI 192.168.201.159@tcp [ 3691.217211] Key type .llcrypt unregistered [ 3691.218382] Key type ._llcrypt unregistered [ 3705.305200] Key type ._llcrypt registered [ 3705.306973] Key type .llcrypt registered [ 3705.347555] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3705.700992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3705.750125] alg: No test for adler32 (adler32-zlib) [ 3706.625476] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3706.731311] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3708.319101] Key type lgssc registered [ 3708.661326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3711.196284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3711.201546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.283656] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3712.292205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3712.342473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.631161] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3716.358959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.388302] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3716.401095] Lustre: 225657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3716.499397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3717.816189] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3720.128714] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3721.872455] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3724.005991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3724.073547] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3724.139640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.521318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3726.093477] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3728.419774] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3729.960842] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3737.560275] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3744.102794] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3748.589456] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3765.700807] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3765.702969] Lustre: Skipped 1 previous similar message [ 3785.260246] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3785.262364] Lustre: Skipped 2 previous similar messages [ 3819.512990] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3819.516363] Lustre: Skipped 3 previous similar messages [ 3890.885680] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3890.888188] Lustre: Skipped 8 previous similar messages [ 3894.756370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3894.757084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.758473] LustreError: Skipped 1 previous similar message [ 3894.762583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3894.764555] Lustre: Skipped 1 previous similar message [ 3898.441975] Lustre: server umount lustre-OST0000 complete [ 3899.872414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.872922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3899.876956] Lustre: Skipped 2 previous similar messages [ 3899.880197] Lustre: Skipped 1 previous similar message [ 3904.992052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.994753] Lustre: Skipped 1 previous similar message [ 3905.751752] Lustre: server umount lustre-MDT0000 complete [ 3906.943960] LustreError: 225647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885195 with bad export cookie 9069453648472079602 [ 3906.947027] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3906.947531] LustreError: 225647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3907.059638] Lustre: server umount lustre-MDT0001 complete [ 3910.448898] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3911.456076] Key type lgssc unregistered [ 3911.584349] LNet: 230441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3911.587821] LNetError: 230441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3911.599283] LNet: Removed LNI 192.168.201.159@tcp [ 3911.907711] Key type .llcrypt unregistered [ 3911.908791] Key type ._llcrypt unregistered [ 3920.784778] Key type ._llcrypt registered [ 3920.785737] Key type .llcrypt registered [ 3920.820134] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3921.097067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3921.109561] alg: No test for adler32 (adler32-zlib) [ 3921.952241] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3922.032862] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3923.607139] Key type lgssc registered [ 3923.960688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3926.571912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3926.578776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3927.712632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.007635] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3931.659571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.785751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3933.091129] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3935.350747] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3937.105416] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3939.271987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3939.391941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3941.230957] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3943.617509] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3952.738825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3958.244737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.245570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3958.246466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3958.248078] Lustre: Skipped 1 previous similar message [ 3963.360485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3963.362582] Lustre: Skipped 2 previous similar messages [ 3964.054712] Lustre: server umount lustre-OST0000 complete [ 3967.967816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3967.969837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.973393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3971.343783] Lustre: server umount lustre-MDT0000 complete [ 3972.513033] LustreError: 234424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885261 with bad export cookie 243895957150752949 [ 3972.515794] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3972.517576] LustreError: 234424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3972.627580] Lustre: server umount lustre-MDT0001 complete [ 3976.140807] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 3977.194164] Key type lgssc unregistered [ 3977.316381] LNet: 235527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3977.318702] LNetError: 235527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3977.328264] LNet: Removed LNI 192.168.201.159@tcp [ 3977.624682] Key type .llcrypt unregistered [ 3977.625713] Key type ._llcrypt unregistered [ 3985.812760] Key type ._llcrypt registered [ 3985.813621] Key type .llcrypt registered [ 3985.846602] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3986.260372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3986.268148] alg: No test for adler32 (adler32-zlib) [ 3987.121442] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3987.215751] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 3988.799136] Key type lgssc registered [ 3989.166443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4255.512571] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 4256.659610] Key type lgssc unregistered [ 4256.794509] LNet: 237521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4256.797160] LNetError: 237521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4256.807398] LNet: Removed LNI 192.168.201.159@tcp [ 4257.127329] Key type .llcrypt unregistered [ 4257.128841] Key type ._llcrypt unregistered [ 4265.645547] Key type ._llcrypt registered [ 4265.646577] Key type .llcrypt registered [ 4265.691178] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 4272.331027] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 4272.769389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4272.828317] alg: No test for adler32 (adler32-zlib) [ 4273.699819] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 4273.805531] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 4275.391154] Key type lgssc registered [ 4275.734716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4285.533101] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3969 sec ========== 12:39:33 (1788885573) [ 4286.087000] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:39:34 (1788885574) === [ 4287.208197] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:39:35 (1788885575) === [ 4300.479247] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 4301.561363] Key type lgssc unregistered [ 4301.682336] LNet: 242720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4301.686107] LNetError: 242720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4301.696279] LNet: Removed LNI 192.168.201.159@tcp [ 4301.990370] Key type .llcrypt unregistered [ 4301.991615] Key type ._llcrypt unregistered