[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 553281056 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002373] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007713] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010154] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012047] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.020613] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024755] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027085] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028135] *** VALIDATE tmpfs *** [ 0.030075] *** VALIDATE proc *** [ 0.031270] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033321] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037035] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041473] debug: unmapping init [mem 0xffffffff9a659000-0xffffffff9a660fff] [ 0.043183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044737] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045028] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047015] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051016] ... event mask: 000000070000000f [ 0.053284] rcu: Hierarchical SRCU implementation. [ 0.055669] smp: Bringing up secondary CPUs ... [ 0.056599] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.060406] smp: Brought up 1 node, 4 CPUs [ 0.062011] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.225389] node 0 deferred pages initialised in 160ms [ 0.227128] devtmpfs: initialized [ 0.229058] x86/mm: Memory block size: 128MB [ 0.231997] gcov: version magic: 0x41383552 [ 0.234295] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.235176] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.236365] pinctrl core: initialized pinctrl subsystem [ 0.237276] [ 0.237824] ************************************************************* [ 0.238014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239016] ** ** [ 0.240015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.241018] ** ** [ 0.242015] ** This means that this kernel is built to expose internal ** [ 0.243014] ** IOMMU data structures, which may compromise security on ** [ 0.244014] ** your system. ** [ 0.245016] ** ** [ 0.246013] ** If you see this message and you are not debugging the ** [ 0.247015] ** kernel, report this immediately to your vendor! ** [ 0.248016] ** ** [ 0.249012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.250139] ************************************************************* [ 0.251666] NET: Registered protocol family 16 [ 0.252449] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.253096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.254067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.255610] cpuidle: using governor menu [ 0.257387] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.258596] PCI: Using configuration type 1 for base access [ 0.260128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.270126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.272030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.275141] cryptd: max_cpu_qlen set to 1000 [ 0.278184] ACPI: Added _OSI(Module Device) [ 0.279012] ACPI: Added _OSI(Processor Device) [ 0.280011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.286000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.292580] ACPI: Interpreter enabled [ 0.294054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.296011] ACPI: Using IOAPIC for interrupt routing [ 0.297106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.301447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.312438] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.315053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.317018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.321111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.326515] acpiphp: Slot [2] registered [ 0.328071] acpiphp: Slot [5] registered [ 0.330112] acpiphp: Slot [6] registered [ 0.331087] acpiphp: Slot [7] registered [ 0.332083] acpiphp: Slot [8] registered [ 0.334101] acpiphp: Slot [9] registered [ 0.335118] acpiphp: Slot [10] registered [ 0.337101] acpiphp: Slot [3] registered [ 0.338077] acpiphp: Slot [4] registered [ 0.340089] acpiphp: Slot [11] registered [ 0.341092] acpiphp: Slot [12] registered [ 0.343088] acpiphp: Slot [13] registered [ 0.344099] acpiphp: Slot [14] registered [ 0.346116] acpiphp: Slot [15] registered [ 0.347071] acpiphp: Slot [16] registered [ 0.348092] acpiphp: Slot [17] registered [ 0.350078] acpiphp: Slot [18] registered [ 0.351130] acpiphp: Slot [19] registered [ 0.352073] acpiphp: Slot [20] registered [ 0.354111] acpiphp: Slot [21] registered [ 0.355176] acpiphp: Slot [22] registered [ 0.357104] acpiphp: Slot [23] registered [ 0.358088] acpiphp: Slot [24] registered [ 0.360146] acpiphp: Slot [25] registered [ 0.361086] acpiphp: Slot [26] registered [ 0.363074] acpiphp: Slot [27] registered [ 0.364100] acpiphp: Slot [28] registered [ 0.366093] acpiphp: Slot [29] registered [ 0.367091] acpiphp: Slot [30] registered [ 0.368168] acpiphp: Slot [31] registered [ 0.370091] PCI host bridge to bus 0000:00 [ 0.371014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.373014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.375016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.378019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.380029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.383029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.385182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388073] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.391305] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.401015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.406532] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.409015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.411023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.413020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416469] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.420086] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.426216] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.433020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.446022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.450013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.454955] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.460017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.465015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.477017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.487879] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.492013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.510019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.520227] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.532018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.543014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.582018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.604855] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.619017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.633025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.690021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.709185] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.726032] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.742024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.791017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.811193] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.838018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.866020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.930023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.949629] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.951295] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.953333] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.955295] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.957176] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.962063] iommu: Default domain type: Passthrough [ 0.963408] SCSI subsystem initialized [ 0.964111] ACPI: bus type USB registered [ 0.965089] usbcore: registered new interface driver usbfs [ 0.967054] usbcore: registered new interface driver hub [ 0.968087] usbcore: registered new device driver usb [ 0.970139] pps_core: LinuxPPS API ver. 1 registered [ 0.971008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.973056] PTP clock support registered [ 0.975198] EDAC MC: Ver: 3.0.0 [ 0.977134] PCI: Using ACPI for IRQ routing [ 0.978865] NetLabel: Initializing [ 0.980012] NetLabel: domain hash size = 128 [ 0.981006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.982076] NetLabel: unlabeled traffic allowed by default [ 0.984148] vgaarb: loaded [ 0.986257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.988014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.993505] clocksource: Switched to clocksource kvm-clock [ 1.105135] VFS: Disk quotas dquot_6.6.0 [ 1.106625] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.110275] *** VALIDATE ramfs *** [ 1.112162] *** VALIDATE hugetlbfs *** [ 1.114322] pnp: PnP ACPI init [ 1.117628] pnp: PnP ACPI: found 6 devices [ 1.139120] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.143542] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.146544] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.149278] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.152561] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.155672] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.159232] NET: Registered protocol family 2 [ 1.162605] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.168695] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.173594] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.179652] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.182044] TCP: Hash tables configured (established 65536 bind 65536) [ 1.184021] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.185953] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.187839] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.189707] NET: Registered protocol family 1 [ 1.191951] RPC: Registered named UNIX socket transport module. [ 1.193292] RPC: Registered udp transport module. [ 1.194321] RPC: Registered tcp transport module. [ 1.195683] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.197648] NET: Registered protocol family 44 [ 1.198863] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.200845] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.202558] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.204057] PCI: CLS 0 bytes, default 64 [ 1.205140] Unpacking initramfs... [ 2.578398] debug: unmapping init [mem 0xffff8a7a3cc54000-0xffff8a7a3ffbffff] [ 2.581599] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.583334] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.585461] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.136896] Initialise system trusted keyrings [ 3.138342] Key type blacklist registered [ 3.139768] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.148624] zbud: loaded [ 3.151397] *** VALIDATE nfs *** [ 3.152433] *** VALIDATE nfs4 *** [ 3.154087] pstore: using deflate compression [ 3.157305] Platform Keyring initialized [ 3.263500] NET: Registered protocol family 38 [ 3.265171] Key type asymmetric registered [ 3.266754] Asymmetric key parser 'x509' registered [ 3.268691] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.271787] io scheduler mq-deadline registered [ 3.273221] io scheduler kyber registered [ 3.275221] io scheduler bfq registered [ 3.277203] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.280164] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.283028] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.285674] ACPI: Power Button [PWRF] [ 3.292446] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.299494] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.322151] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.332169] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.360324] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.389500] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.418113] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.423249] Non-volatile memory driver v1.3 [ 3.426078] Linux agpgart interface v0.103 [ 3.469266] virtio_blk virtio1: [vda] 145928 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.472660] vda: detected capacity change from 0 to 74715136 [ 3.499409] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.502664] vdb: detected capacity change from 0 to 1073741824 [ 3.531470] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.535410] vdc: detected capacity change from 0 to 2621440000 [ 3.584239] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.587772] vdd: detected capacity change from 0 to 2621440000 [ 3.630707] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.634369] vde: detected capacity change from 0 to 4294967296 [ 3.680327] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.683339] vdf: detected capacity change from 0 to 4294967296 [ 3.710263] libphy: Fixed MDIO Bus: probed [ 3.720480] usbcore: registered new interface driver usbserial_generic [ 3.722706] usbserial: USB Serial support registered for generic [ 3.725135] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.729687] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.731493] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.733692] mousedev: PS/2 mouse device common for all mice [ 3.736844] rtc_cmos 00:05: RTC can wake from S4 [ 3.740644] rtc_cmos 00:05: registered as rtc0 [ 3.742147] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.743600] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.749711] intel_pstate: CPU model not supported [ 3.753392] hid: raw HID events driver (C) Jiri Kosina [ 3.753500] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.755760] usbcore: registered new interface driver usbhid [ 3.760713] usbhid: USB HID core driver [ 3.762812] drop_monitor: Initializing network drop monitor service [ 3.765822] Initializing XFRM netlink socket [ 3.768156] NET: Registered protocol family 10 [ 3.768280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.772109] Segment Routing with IPv6 [ 3.774027] NET: Registered protocol family 17 [ 3.776081] mpls_gso: MPLS GSO support [ 3.782371] RAS: Correctable Errors collector initialized. [ 3.784247] AVX version of gcm_enc/dec engaged. [ 3.785709] AES CTR mode by8 optimization enabled [ 3.856950] sched_clock: Marking stable (3856879617, 0)->(4764068226, -907188609) [ 3.861135] registered taskstats version 1 [ 3.863468] Loading compiled-in X.509 certificates [ 3.865742] zswap: loaded using pool lzo/zbud [ 3.889129] Key type big_key registered [ 3.900909] Key type encrypted registered [ 3.902361] ima: No TPM chip found, activating TPM-bypass! [ 3.904536] ima: Allocated hash algorithm: sha1 [ 3.906659] ima: No architecture policies found [ 3.908779] evm: Initialising EVM extended attributes: [ 3.911148] evm: security.selinux [ 3.912659] evm: security.ima [ 3.914094] evm: security.capability [ 3.915669] evm: HMAC attrs: 0x1 [ 3.918570] rtc_cmos 00:05: setting system clock to 2026-07-30 05:17:00 UTC (1785388620) [ 3.926102] debug: unmapping init [mem 0xffffffff9b603000-0xffffffff9b7fffff] [ 3.929309] debug: unmapping init [mem 0xffffffff9a382000-0xffffffff9a658fff] [ 3.938102] Write protecting the kernel read-only data: 28672k [ 3.941633] debug: unmapping init [mem 0xffffffff98a03000-0xffffffff98bfffff] [ 3.944408] debug: unmapping init [mem 0xffffffff99314000-0xffffffff993fffff] [ 3.978551] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.986733] systemd[1]: Detected virtualization kvm. [ 3.988767] systemd[1]: Detected architecture x86-64. [ 3.990682] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.021627] systemd[1]: No hostname configured. [ 4.023637] systemd[1]: Set hostname to . [ 4.025828] random: systemd: uninitialized urandom read (16 bytes read) [ 4.028522] systemd[1]: Initializing machine ID from random generator. [ 4.119060] random: ln: uninitialized urandom read (6 bytes read) [ 4.215289] random: systemd: uninitialized urandom read (16 bytes read) [ 4.217843] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.225332] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.237576] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.919140] device-mapper: uevent: version 1.0.3 [ 4.922188] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. [ 5.150924] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.641240] virtio_net virtio0 ens2: renamed from eth0 [ 5.691202] scsi host0: ata_piix [ 5.698216] scsi host1: ata_piix [ 5.700112] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.703340] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.331613] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.420453] random: crng init done [ 10.422387] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.864725] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.971441] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.237517] SELinux: Disabled at runtime. [ 12.305464] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.312656] systemd[1]: Detected virtualization kvm. [ 12.314049] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.784251] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.786104] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.789684] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.791890] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.793732] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.802071] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.806357] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ 12.943075] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.235812] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.531856] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.549350] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.658039] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.680358] EDAC sbridge: Ver: 1.1.2 [ 15.443992] Key type dns_resolver registered [ 15.748133] NFS: Registering the id_resolver key type [ 15.749829] Key type id_resolver registered [ 15.751400] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg211-server login: [ 68.319937] hrtimer: interrupt took 2998938 ns [ 97.235398] libcfs: loading out-of-tree module taints kernel. [ 97.259423] Key type ._llcrypt registered [ 97.264849] Key type .llcrypt registered [ 97.469834] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 128.608988] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 131.731039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.767854] alg: No test for adler32 (adler32-zlib) [ 134.362391] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 136.013640] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 138.066099] Key type lgssc registered [ 141.182910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 167.190074] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 241.904102] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 273.060113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 273.127666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.710935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 274.825995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 274.999259] Lustre: lustre-MDT0000: new disk, initializing [ 275.185891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 275.219828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 282.797664] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 302.108099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 302.222388] Lustre: 6545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 302.257402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 302.262482] Lustre: Skipped 1 previous similar message [ 302.405314] Lustre: lustre-MDT0001: new disk, initializing [ 302.522627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 302.543420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 302.550274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 309.414379] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 315.821643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 329.831419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 330.334285] Lustre: lustre-OST0000: new disk, initializing [ 330.337782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 330.350830] Lustre: 8484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 330.438309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.036035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 338.055086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 338.189885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 339.677331] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 361.516351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 361.757221] Lustre: lustre-OST0001: new disk, initializing [ 361.768322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 361.776110] Lustre: 9558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 361.918438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 369.285363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 369.298499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 369.470855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 371.091663] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 386.597206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 395.453465] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 408.592889] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing check_logdir /tmp/testlogs/ [ 415.739905] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing yml_node [ 423.480393] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 426.413087] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 430.250265] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 432.522189] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 30 01:24:06 EDT 2026 [ 458.922964] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 461.359856] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 481.768966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 481.784100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.814242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.296375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.302271] Lustre: Skipped 2 previous similar messages [ 486.869197] Lustre: server umount lustre-MDT0000 complete [ 491.988287] LustreError: 10378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389108 with bad export cookie 17861652778080437972 [ 491.991559] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.000448] LustreError: 10378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 492.514045] LustreError: 8493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.519870] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.534174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 492.534183] Lustre: Skipped 4 previous similar messages [ 492.551709] LustreError: 8493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 492.627678] Lustre: Skipped 1 previous similar message [ 496.551356] LustreError: 12312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.552607] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 496.591969] LustreError: 12312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 496.607728] Lustre: Skipped 2 previous similar messages [ 501.669428] LustreError: 11129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.669553] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 501.690023] LustreError: 11129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 501.699874] Lustre: Skipped 1 previous similar message [ 506.336601] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 506.857709] LustreError: 8493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.085081] Lustre: server umount lustre-MDT0001 complete [ 512.940896] Lustre: server umount lustre-OST0000 complete [ 518.863278] Lustre: server umount lustre-OST0001 complete [ 530.408632] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 547.122319] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 606.232425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.702329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 606.736979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 606.869905] Lustre: lustre-MDT0000: new disk, initializing [ 606.981736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.006504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 614.557860] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 631.006679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.235573] Lustre: 15152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 631.386761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 631.395979] Lustre: Skipped 1 previous similar message [ 631.508445] Lustre: lustre-MDT0001: new disk, initializing [ 631.717585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 631.818181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 631.845786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 638.507582] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 651.960335] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 661.664954] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 675.479807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 676.146273] Lustre: lustre-OST0000: new disk, initializing [ 676.156597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 676.173693] Lustre: 17122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 676.376307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.345859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 677.368865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 677.665325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 685.885296] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 700.725942] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 708.415851] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 708.823835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 715.150630] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 715.474883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 718.835280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 718.844065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.856883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 723.443382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 723.455459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.064524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 729.085861] Lustre: Skipped 3 previous similar messages [ 733.152143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 733.527960] Lustre: server umount lustre-OST0000 complete [ 744.424898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 744.425694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.314855] Lustre: server umount lustre-MDT0000 complete [ 749.541471] LustreError: 16208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.594492] LustreError: 16208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 753.154048] LustreError: 15143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389369 with bad export cookie 17861652778080439505 [ 753.159708] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.164585] LustreError: 15143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 754.157358] Lustre: server umount lustre-MDT0001 complete [ 759.379876] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:29:32 (1785389372) [ 772.407485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.224420] LustreError: 19393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.461462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 780.643420] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 793.057800] LustreError: 19394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 793.096635] LustreError: 19394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 794.228965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.704053] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 812.387324] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 821.011197] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 832.054321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 840.571878] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 850.382765] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 863.714801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 863.717295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.747444] Lustre: Skipped 1 previous similar message [ 863.766842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 863.776374] Lustre: Skipped 3 previous similar messages [ 867.096538] Lustre: server umount lustre-OST0000 complete [ 880.112481] Lustre: server umount lustre-MDT0000 complete [ 883.681576] LustreError: 20221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.715979] LustreError: 20221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 887.226171] LustreError: 19375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389503 with bad export cookie 17861652778080440380 [ 887.278351] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 888.224253] Lustre: server umount lustre-MDT0001 complete [ 914.568129] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 919.701688] Key type lgssc unregistered [ 920.190969] LNet: 22964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.199837] LNetError: 22964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.211857] LNet: Removed LNI 192.168.202.111@tcp [ 922.109848] Key type .llcrypt unregistered [ 922.111040] Key type ._llcrypt unregistered [ 956.971372] Key type ._llcrypt registered [ 956.979285] Key type .llcrypt registered [ 957.134791] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 958.882365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 959.050852] alg: No test for adler32 (adler32-zlib) [ 960.397630] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 960.833447] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 962.643941] Key type lgssc registered [ 964.754949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 976.997851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 977.024910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.884401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.511601] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 989.153473] LustreError: 24379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.171329] LustreError: 24379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 994.275656] LustreError: 24380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.799675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.525403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.268961] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1015.973261] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1024.415750] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1034.067296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1034.776382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.582722] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1049.207650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1055.861860] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1085.418801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.485902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1085.505756] Lustre: Skipped 1 previous similar message [ 1090.535789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.543804] Lustre: Skipped 1 previous similar message [ 1090.958538] Lustre: server umount lustre-OST0000 complete [ 1100.777251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.836757] Lustre: Skipped 1 previous similar message [ 1100.844127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1105.003355] Lustre: server umount lustre-MDT0000 complete [ 1105.890199] LustreError: 24374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.916639] LustreError: 24374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1111.017978] LustreError: 24380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1111.031439] LustreError: 24380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1111.431517] LustreError: 27621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389728 with bad export cookie 9656238602368608056 [ 1111.433933] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1111.445943] LustreError: 27621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1111.727624] Lustre: server umount lustre-MDT0001 complete [ 1126.222604] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1130.264637] Key type lgssc unregistered [ 1130.788393] LNet: 28375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.800450] LNetError: 28375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1130.826565] LNet: Removed LNI 192.168.202.111@tcp [ 1132.379658] Key type .llcrypt unregistered [ 1132.381573] Key type ._llcrypt unregistered [ 1169.853813] Key type ._llcrypt registered [ 1169.859175] Key type .llcrypt registered [ 1170.085768] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1171.727259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1171.989663] alg: No test for adler32 (adler32-zlib) [ 1173.524761] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 1174.173086] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1175.952207] Key type lgssc registered [ 1177.616301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1189.446992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1189.508679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.202608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.217570] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1200.691094] LustreError: 29795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.699278] LustreError: 29795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1205.756508] LustreError: 29796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.855841] LustreError: 29795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.120354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.819610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.156534] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1235.084210] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1242.306701] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1261.411172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1261.943331] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1262.105768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.217970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1271.451190] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1282.513596] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1297.897733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1297.905741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.909112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.917441] Lustre: Skipped 1 previous similar message [ 1297.932160] Lustre: Skipped 1 previous similar message [ 1303.013770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1303.027136] Lustre: Skipped 2 previous similar messages [ 1307.104145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1307.522984] Lustre: server umount lustre-OST0000 complete [ 1313.250758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1313.251281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.254870] LustreError: Skipped 1 previous similar message [ 1313.269029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.273720] Lustre: Skipped 1 previous similar message [ 1318.371085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1318.385761] Lustre: Skipped 1 previous similar message [ 1318.874980] Lustre: server umount lustre-MDT0000 complete [ 1323.492731] LustreError: 29795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.526226] LustreError: 29795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1324.176308] LustreError: 29777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785389940 with bad export cookie 1992309460757450776 [ 1324.198639] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1324.212584] LustreError: 29777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1324.886906] Lustre: server umount lustre-MDT0001 complete [ 1349.577878] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1355.156440] Key type lgssc unregistered [ 1355.870628] LNet: 33702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.890971] LNetError: 33702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.943580] LNet: Removed LNI 192.168.202.111@tcp [ 1358.533894] Key type .llcrypt unregistered [ 1358.547774] Key type ._llcrypt unregistered [ 1394.138812] Key type ._llcrypt registered [ 1394.141831] Key type .llcrypt registered [ 1394.244972] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1395.442703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1395.674391] alg: No test for adler32 (adler32-zlib) [ 1396.998703] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 1397.357399] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1399.080108] Key type lgssc registered [ 1401.258593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1412.306394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1412.328783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.099125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.251767] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1424.355381] LustreError: 35118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.372160] LustreError: 35118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1429.476963] LustreError: 35117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.903715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.720684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1442.284662] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1455.052396] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.756805] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1475.025802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1475.682385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.960157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1485.229241] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1497.531571] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1514.991114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.011789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1515.031856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1515.058632] Lustre: Skipped 1 previous similar message [ 1518.181567] Lustre: server umount lustre-OST0000 complete [ 1525.219871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1525.222804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1525.239192] Lustre: Skipped 2 previous similar messages [ 1525.254530] Lustre: Skipped 1 previous similar message [ 1530.342783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1530.642772] Lustre: server umount lustre-MDT0000 complete [ 1534.436334] LustreError: 35117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.476070] LustreError: 35117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1536.001342] LustreError: 35099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390152 with bad export cookie 17668457886627147918 [ 1536.006905] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1536.021491] LustreError: 35099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1536.548258] Lustre: server umount lustre-MDT0001 complete [ 1551.810178] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1557.193920] Key type lgssc unregistered [ 1557.741219] LNet: 38683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.749288] LNetError: 38683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.795246] LNet: Removed LNI 192.168.202.111@tcp [ 1559.476138] Key type .llcrypt unregistered [ 1559.477564] Key type ._llcrypt unregistered [ 1598.265345] Key type ._llcrypt registered [ 1598.271217] Key type .llcrypt registered [ 1598.595858] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1600.522435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.594360] alg: No test for adler32 (adler32-zlib) [ 1602.761932] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 1603.568169] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1605.568394] Key type lgssc registered [ 1608.964742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1622.614725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1622.637795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.990877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.254546] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1634.281720] LustreError: 40099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.309968] LustreError: 40099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1639.417893] LustreError: 40098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.434976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.523020] LustreError: 40099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.156021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1652.167696] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1662.945607] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1670.418506] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1680.500469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1681.067275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.930053] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1694.507758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1702.324718] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1709.896487] Lustre: lustre-OST0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 1710.051989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.053380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.998716] Lustre: lustre-OST0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 1715.014991] Lustre: Skipped 2 previous similar messages [ 1720.115339] Lustre: lustre-OST0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 1720.125522] Lustre: Skipped 2 previous similar messages [ 1723.360110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1723.634979] Lustre: server umount lustre-OST0000 complete [ 1751.010830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.025308] Lustre: Skipped 1 previous similar message [ 1751.035444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.039748] Lustre: Skipped 2 previous similar messages [ 1755.639060] Lustre: server umount lustre-MDT0000 complete [ 1756.130914] LustreError: 40092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.166528] LustreError: 40092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1761.259254] LustreError: 40099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.323284] LustreError: 40099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1761.903100] LustreError: 40079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390378 with bad export cookie 10496676013552213743 [ 1761.932114] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.948360] LustreError: 40079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1769.292420] Lustre: server umount lustre-MDT0001 complete [ 1791.032319] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 1795.095517] Key type lgssc unregistered [ 1795.557701] LNet: 43834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.566987] LNetError: 43834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.608301] LNet: Removed LNI 192.168.202.111@tcp [ 1796.967406] Key type .llcrypt unregistered [ 1796.970380] Key type ._llcrypt unregistered [ 1831.780215] Key type ._llcrypt registered [ 1831.785312] Key type .llcrypt registered [ 1832.055679] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 1834.132062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.154263] alg: No test for adler32 (adler32-zlib) [ 1835.715477] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 1836.214261] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 1838.104730] Key type lgssc registered [ 1839.760674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1849.948302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1849.972072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.605834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.390640] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1862.126755] LustreError: 45251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.160885] LustreError: 45251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1867.256904] LustreError: 45252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.182916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.556678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1877.948956] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1890.622320] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1899.674035] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1908.984844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1909.550233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.872434] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 1921.856645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1927.900512] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1937.377179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1937.381081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.385595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.408136] Lustre: Skipped 2 previous similar messages [ 1939.795039] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.11@tcp (stopping) [ 1939.812195] Lustre: Skipped 2 previous similar messages [ 1942.045593] Lustre: server umount lustre-MDT0000 complete [ 1942.499556] LustreError: 45980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.526193] LustreError: 45980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1944.877594] LustreError: 45246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.837079] LustreError: 45234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785390563 with bad export cookie 14279733227102231383 [ 1946.847424] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.862628] LustreError: 45234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1947.619936] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.652593] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1949.997494] LustreError: 46196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.009538] LustreError: 46196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1950.021560] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.11@tcp (stopping) [ 1953.616921] Lustre: server umount lustre-MDT0001 complete [ 1970.701794] Lustre: server umount lustre-OST0000 complete [ 2005.709563] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2011.036271] Key type lgssc unregistered [ 2011.795715] LNet: 49134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.808743] LNetError: 49134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.852597] LNet: Removed LNI 192.168.202.111@tcp [ 2014.093625] Key type .llcrypt unregistered [ 2014.099360] Key type ._llcrypt unregistered [ 2051.799853] Key type ._llcrypt registered [ 2051.801772] Key type .llcrypt registered [ 2052.012646] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2054.416772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.509128] alg: No test for adler32 (adler32-zlib) [ 2055.938869] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 2056.514515] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2058.264832] Key type lgssc registered [ 2061.169268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.785743] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2071.831654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.648367] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2092.768391] LustreError: 50524:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 2119.585036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.106650] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2142.076518] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2309.288489] Lustre: server umount lustre-OST0000 complete [ 2334.172925] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2340.582342] Key type lgssc unregistered [ 2341.204460] LNet: 52204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.208992] LNetError: 52204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2341.251315] LNet: Removed LNI 192.168.202.111@tcp [ 2343.355768] Key type .llcrypt unregistered [ 2343.358582] Key type ._llcrypt unregistered [ 2383.220141] Key type ._llcrypt registered [ 2383.224110] Key type .llcrypt registered [ 2383.459793] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2385.343223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2385.542784] alg: No test for adler32 (adler32-zlib) [ 2386.808881] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 2387.231191] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2389.048483] Key type lgssc registered [ 2391.397552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2404.236720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2404.293794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.054821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.666562] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2415.597702] LustreError: 53623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.642844] LustreError: 53623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2420.707715] LustreError: 53622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.838313] LustreError: 53623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.892367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.691490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2436.562160] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2447.419508] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2455.452490] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2465.491583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2465.991037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.462869] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2476.292858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2485.058611] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2490.851123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2490.875992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.943118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.880589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2491.899719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.906547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.997212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2497.017680] Lustre: Skipped 1 previous similar message [ 2502.112252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2502.114391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2502.161228] Lustre: Skipped 2 previous similar messages [ 2502.611360] Lustre: server umount lustre-OST0000 complete [ 2511.841514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2511.866079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.898888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.826956] Lustre: server umount lustre-MDT0000 complete [ 2517.492594] LustreError: 53623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.536165] LustreError: 53623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2520.858107] LustreError: 53601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391137 with bad export cookie 4855279636981905891 [ 2520.869735] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2520.877029] LustreError: 53601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2522.170279] Lustre: server umount lustre-MDT0001 complete [ 2539.502887] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2544.036569] Key type lgssc unregistered [ 2544.526356] LNet: 57161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.539477] LNetError: 57161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.576641] LNet: Removed LNI 192.168.202.111@tcp [ 2547.075667] Key type .llcrypt unregistered [ 2547.077543] Key type ._llcrypt unregistered [ 2582.516837] Key type ._llcrypt registered [ 2582.519842] Key type .llcrypt registered [ 2582.675919] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2584.900597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2584.930945] alg: No test for adler32 (adler32-zlib) [ 2586.106232] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 2586.457527] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2588.248201] Key type lgssc registered [ 2589.803788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2600.838345] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2600.866683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.544364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2621.667632] LustreError: 58550:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 2648.416238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.137307] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2678.413747] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2688.463908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.045764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.210090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2694.116609] LustreError: 58574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.145481] LustreError: 58574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2697.017554] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2699.275022] LustreError: 58575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.355287] LustreError: 58574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.429320] LustreError: 58575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.476298] LustreError: 58575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2712.576260] LustreError: 59605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.621889] LustreError: 59605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2716.439397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.246683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2726.524976] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2738.043574] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2748.296861] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2752.994350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2753.008947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.016552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2753.034114] Lustre: Skipped 1 previous similar message [ 2753.076823] Lustre: Skipped 1 previous similar message [ 2758.117632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2758.829029] Lustre: server umount lustre-OST0000 complete [ 2788.833684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2788.855201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.885562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2788.893287] Lustre: Skipped 1 previous similar message [ 2792.942808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.947029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2792.988196] Lustre: Skipped 1 previous similar message [ 2795.013419] Lustre: server umount lustre-MDT0000 complete [ 2798.055487] LustreError: 59608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.078109] LustreError: 59608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2802.108493] LustreError: 58558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391418 with bad export cookie 13414803491239392980 [ 2802.122570] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2802.129674] LustreError: 58558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2808.941457] Lustre: server umount lustre-MDT0001 complete [ 2824.286481] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 2828.966167] Key type lgssc unregistered [ 2829.501420] LNet: 62323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2829.508202] LNetError: 62323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2829.575467] LNet: Removed LNI 192.168.202.111@tcp [ 2831.499585] Key type .llcrypt unregistered [ 2831.500981] Key type ._llcrypt unregistered [ 2865.762453] Key type ._llcrypt registered [ 2865.763955] Key type .llcrypt registered [ 2865.942652] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 2867.934753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2867.983853] alg: No test for adler32 (adler32-zlib) [ 2869.333906] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 2870.035440] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 2871.920425] Key type lgssc registered [ 2874.436250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2888.824257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2888.863108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.954298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.488848] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2900.451837] LustreError: 63759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.469211] LustreError: 63759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2905.569266] LustreError: 63760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.709396] LustreError: 63759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.395041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.089664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2920.316415] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2930.293184] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2939.609147] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2953.378925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.243394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2960.473011] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2970.373598] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 2986.033783] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3006.448721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.449344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3006.459389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3006.462682] Lustre: Skipped 1 previous similar message [ 3011.559180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3011.570121] Lustre: Skipped 1 previous similar message [ 3016.675449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3016.682847] Lustre: Skipped 1 previous similar message [ 3020.256127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3020.737303] Lustre: server umount lustre-OST0000 complete [ 3026.913342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.920703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3026.939965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3026.939973] Lustre: Skipped 1 previous similar message [ 3032.034382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3032.041610] Lustre: Skipped 2 previous similar messages [ 3032.671763] Lustre: server umount lustre-MDT0000 complete [ 3037.154496] LustreError: 63760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.186053] LustreError: 63760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3038.365552] LustreError: 63741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391654 with bad export cookie 5271823771058704703 [ 3038.390686] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3039.264628] Lustre: server umount lustre-MDT0001 complete [ 3059.876841] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3064.163443] Key type lgssc unregistered [ 3064.779640] LNet: 67351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3064.807192] LNetError: 67351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3064.837208] LNet: Removed LNI 192.168.202.111@tcp [ 3066.727319] Key type .llcrypt unregistered [ 3066.729972] Key type ._llcrypt unregistered [ 3104.649829] Key type ._llcrypt registered [ 3104.653905] Key type .llcrypt registered [ 3104.928263] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3107.073366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3107.123113] alg: No test for adler32 (adler32-zlib) [ 3108.606838] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 3109.176455] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3111.023236] Key type lgssc registered [ 3113.167712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3127.865560] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:09:01 (1785391741) [ 3137.791722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3137.822090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.517931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.028159] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3149.793530] LustreError: 69061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.822240] LustreError: 69061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3154.924413] LustreError: 69060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.040263] LustreError: 69061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.432959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.244916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3168.243890] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3179.520366] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3191.225835] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3205.125508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.834949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.932745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 3217.720572] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3228.409777] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3236.066941] Lustre: Failing over lustre-MDT0000 [ 3236.688989] LustreError: 69056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.023562] Lustre: server umount lustre-MDT0000 complete [ 3237.856864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3237.867588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.936798] LustreError: 69060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.938151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.970472] LustreError: 69060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3249.997385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.231351] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3250.429664] Lustre: *** cfs_fail_loc=135, val=0*** [ 3250.669366] LustreError: 72151:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3250.683918] LustreError: 72151:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.111@tcp: cfg command failed: rc = -2 [ 3250.705045] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3250.705045] [ 3250.718013] LustreError: MGC192.168.202.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3250.735604] LustreError: 72140:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3250.756133] LustreError: 72140:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3250.790551] LustreError: 72140:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3250.806952] LustreError: 72140:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3250.858638] Lustre: server umount lustre-MDT0000 complete [ 3250.864211] LustreError: 72140:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3251.186048] LustreError: 69061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.226187] LustreError: 69061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3260.938079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.129848] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3261.149329] LustreError: 69052:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3261.422144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3261.438493] Lustre: Skipped 2 previous similar messages [ 3261.601161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.672569] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3262.264279] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3267.087970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3267.146396] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3267.240813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 3268.585322] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3282.414981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.416406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3282.436409] Lustre: Skipped 2 previous similar messages [ 3287.531740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3287.550681] Lustre: Skipped 2 previous similar messages [ 3288.633553] Lustre: server umount lustre-OST0000 complete [ 3297.762832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.763100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3297.780675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.970512] Lustre: server umount lustre-MDT0000 complete [ 3302.894709] LustreError: 69061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.925768] LustreError: 69061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3305.039651] LustreError: 69041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785391921 with bad export cookie 15121902970605208784 [ 3305.043638] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.047944] LustreError: 69041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3305.416915] Lustre: server umount lustre-MDT0001 complete [ 3325.916413] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3330.625664] Key type lgssc unregistered [ 3331.194565] LNet: 73958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3331.206319] LNetError: 73958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3331.223858] LNet: Removed LNI 192.168.202.111@tcp [ 3332.331884] Key type .llcrypt unregistered [ 3332.334579] Key type ._llcrypt unregistered [ 3367.233505] Key type ._llcrypt registered [ 3367.237602] Key type .llcrypt registered [ 3367.399607] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3369.274832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3369.471257] alg: No test for adler32 (adler32-zlib) [ 3370.414218] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 3370.569768] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3372.193038] Key type lgssc registered [ 3375.122767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3388.424961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3388.449982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.539431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.062911] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3400.688109] LustreError: 75375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3400.717084] LustreError: 75375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3401.699681] LustreError: 75374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.818404] LustreError: 75375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.993843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.687514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3417.129962] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3427.405650] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3436.195111] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3447.393980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3447.659688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.806267] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3462.040052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3471.195938] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3479.113860] Lustre: Failing over lustre-MDT0000 [ 3480.140167] Lustre: server umount lustre-MDT0000 complete [ 3481.057133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3481.073637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.117335] LustreError: 75374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.594630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.636474] Lustre: Skipped 1 previous similar message [ 3487.715876] LustreError: 75371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3487.737290] LustreError: 75371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3494.568693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3494.674258] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3494.788930] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3494.893499] LustreError: 78477:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 3494.899589] LustreError: 78477:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.111@tcp: cfg command failed: rc = -9 [ 3494.904201] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3494.904201] [ 3494.926411] LustreError: MGC192.168.202.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3494.939263] LustreError: 78466:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3494.948980] LustreError: 78466:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 3494.966828] LustreError: 78466:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3494.980606] LustreError: 78466:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3495.007967] Lustre: server umount lustre-MDT0000 complete [ 3495.012180] LustreError: 78466:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3497.955616] LustreError: 75369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.980848] LustreError: 75369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3505.190374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.395240] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3505.415317] LustreError: 75366:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3505.871719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.945387] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3509.116413] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3511.285609] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3511.370308] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3511.455255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3513.642370] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3526.630223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.641369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.659125] Lustre: Skipped 1 previous similar message [ 3526.659547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3526.678412] Lustre: Skipped 1 previous similar message [ 3531.748172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3531.752286] Lustre: Skipped 1 previous similar message [ 3536.880720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.897133] Lustre: Skipped 1 previous similar message [ 3538.400127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3538.817797] Lustre: server umount lustre-OST0000 complete [ 3547.623443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3547.635955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.662157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.232497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3552.234849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.284512] Lustre: Skipped 1 previous similar message [ 3554.390651] Lustre: server umount lustre-MDT0000 complete [ 3557.624890] LustreError: 75369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.668029] LustreError: 75369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3565.504272] LustreError: 75356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392182 with bad export cookie 18439916270100921846 [ 3565.532627] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3566.510380] Lustre: server umount lustre-MDT0001 complete [ 3590.658638] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3595.520545] Key type lgssc unregistered [ 3596.138526] LNet: 80287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3596.145949] LNetError: 80287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3597.158605] LNet: Removed LNI 192.168.202.111@tcp [ 3597.901527] Key type .llcrypt unregistered [ 3597.903112] Key type ._llcrypt unregistered [ 3626.306869] Key type ._llcrypt registered [ 3626.309286] Key type .llcrypt registered [ 3626.474678] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3626.852343] Key type .llcrypt unregistered [ 3626.855713] Key type ._llcrypt unregistered [ 3667.290000] Key type ._llcrypt registered [ 3667.295238] Key type .llcrypt registered [ 3667.507462] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3669.809520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3669.977810] alg: No test for adler32 (adler32-zlib) [ 3671.519163] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 3672.296540] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3674.097579] Key type lgssc registered [ 3676.184420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3687.826433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3687.881765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.017443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.748740] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3700.200167] LustreError: 82198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3700.240303] LustreError: 82198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3705.334033] LustreError: 82199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.437956] LustreError: 82198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.834261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.484604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3719.713796] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3730.670082] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3739.266556] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3748.408599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3748.778308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.718815] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3760.070239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3772.221620] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3787.750599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3787.757824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3787.782269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3789.984992] Lustre: server umount lustre-OST0000 complete [ 3799.522177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3799.538507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.563142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3801.059400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3801.080446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.177979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.184584] Lustre: Skipped 2 previous similar messages [ 3807.508545] Lustre: server umount lustre-MDT0000 complete [ 3812.326928] LustreError: 82193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.346320] LustreError: 82193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3812.769132] LustreError: 82180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392429 with bad export cookie 11536464056171665593 [ 3812.792516] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3813.773777] Lustre: server umount lustre-MDT0001 complete [ 3830.556828] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 3835.034425] Key type lgssc unregistered [ 3835.399899] LNet: 85758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3835.409780] LNetError: 85758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3836.460361] LNet: Removed LNI 192.168.202.111@tcp [ 3838.018712] Key type .llcrypt unregistered [ 3838.026418] Key type ._llcrypt unregistered [ 3873.427701] Key type ._llcrypt registered [ 3873.429825] Key type .llcrypt registered [ 3873.617584] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 3875.068204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3875.077464] alg: No test for adler32 (adler32-zlib) [ 3876.669641] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 3877.200111] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 3879.026347] Key type lgssc registered [ 3880.619522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3893.474700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3893.511350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.197835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.310170] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3904.484305] LustreError: 87173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.520602] LustreError: 87173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3909.604319] LustreError: 87174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.236062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.742652] LustreError: 87173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.067361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3923.301653] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3938.065220] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3954.239688] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3966.297217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.771563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.670684] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 3987.569216] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3990.341400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 4000.737952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4000.750918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.786651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.795849] Lustre: Skipped 1 previous similar message [ 4005.865732] LustreError: 89000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.901028] LustreError: 89000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4006.101512] Lustre: server umount lustre-OST0000 complete [ 4012.516350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4012.525738] LustreError: Skipped 1 previous similar message [ 4012.529099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.539625] Lustre: Skipped 1 previous similar message [ 4012.545209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4016.097621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4016.119350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.163797] Lustre: server umount lustre-MDT0000 complete [ 4021.218305] LustreError: 89018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.247171] LustreError: 89018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4023.619346] LustreError: 87153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392640 with bad export cookie 4619633007551247355 [ 4023.626225] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.634629] LustreError: 87153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4023.984734] Lustre: server umount lustre-MDT0001 complete [ 4041.882489] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 4046.404815] Key type lgssc unregistered [ 4047.060705] LNet: 90739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4047.076349] LNetError: 90739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4048.103579] LNet: Removed LNI 192.168.202.111@tcp [ 4050.183072] Key type .llcrypt unregistered [ 4050.193676] Key type ._llcrypt unregistered [ 4091.001226] Key type ._llcrypt registered [ 4091.003594] Key type .llcrypt registered [ 4091.246788] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4095.085718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4095.113414] alg: No test for adler32 (adler32-zlib) [ 4096.532400] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 4097.282549] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 4099.112193] Key type lgssc registered [ 4100.997737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4115.585417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4115.656513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.750425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.204381] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4128.230146] LustreError: 92156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.257888] LustreError: 92156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4133.346062] LustreError: 92157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.839118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.479112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4144.006030] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4156.192591] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4167.735400] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4182.811063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.411102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.510250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 4197.532844] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4209.352530] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4238.825941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4238.826602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.844020] LustreError: Skipped 1 previous similar message [ 4238.873159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4238.875865] Lustre: Skipped 1 previous similar message [ 4238.908614] Lustre: Skipped 1 previous similar message [ 4241.309845] Lustre: server umount lustre-OST0000 complete [ 4249.064058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.083474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4254.180980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4254.196824] Lustre: Skipped 2 previous similar messages [ 4255.064760] Lustre: server umount lustre-MDT0000 complete [ 4259.307852] LustreError: 92157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.322485] LustreError: 92157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4261.481051] LustreError: 94969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785392878 with bad export cookie 11335742240367157258 [ 4261.482375] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4261.493835] LustreError: 94969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4262.434801] Lustre: server umount lustre-MDT0001 complete [ 4287.315823] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 4293.830295] Key type lgssc unregistered [ 4294.478469] LNet: 95745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4294.506969] LNetError: 95745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4295.596635] LNet: Removed LNI 192.168.202.111@tcp [ 4297.715198] Key type .llcrypt unregistered [ 4297.722589] Key type ._llcrypt unregistered [ 4329.858198] Key type ._llcrypt registered [ 4329.870397] Key type .llcrypt registered [ 4330.021928] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4331.827474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4331.939959] alg: No test for adler32 (adler32-zlib) [ 4333.061519] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 4333.317630] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 4334.992161] Key type lgssc registered [ 4337.433288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4349.948280] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4350.023203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.728612] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4370.848344] LustreError: 97134:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 4397.601237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4408.338602] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4418.434952] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4426.885173] Lustre: server umount lustre-OST0000 complete [ 4440.123336] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:30:54 (1785393054) [ 4450.061457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.500944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.905930] LustreError: 98775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.922031] LustreError: 98775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4456.466937] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4461.041046] LustreError: 98774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.147196] LustreError: 98775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.015949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4469.829196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4476.657465] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4487.190548] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4496.832962] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4506.971875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4507.616971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.205138] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4521.034197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 4529.007962] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4545.505279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4545.510536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.539337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4546.535654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4546.580257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4546.607742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4550.130923] Lustre: server umount lustre-OST0000 complete [ 4561.890310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.917085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4561.923881] Lustre: Skipped 2 previous similar messages [ 4565.344622] Lustre: server umount lustre-MDT0000 complete [ 4567.034915] LustreError: 98775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.060268] LustreError: 98775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4571.622228] LustreError: 98755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785393188 with bad export cookie 2825421086366878060 [ 4571.638703] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4571.648539] LustreError: 98755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4572.753239] Lustre: server umount lustre-MDT0001 complete [ 4590.630133] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 4595.413859] Key type lgssc unregistered [ 4595.759603] LNet: 102473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4595.762554] LNetError: 102473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4595.776642] LNet: Removed LNI 192.168.202.111@tcp [ 4596.593242] Key type .llcrypt unregistered [ 4596.595323] Key type ._llcrypt unregistered [ 4664.960646] Key type ._llcrypt registered [ 4664.964359] Key type .llcrypt registered [ 4665.118663] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4667.648208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4667.749863] alg: No test for adler32 (adler32-zlib) [ 4669.155874] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 4669.874784] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 4671.656176] Key type lgssc registered [ 4672.562670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4687.520587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4687.552337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4689.071331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.829611] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4698.605968] LustreError: 104669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.614252] LustreError: 104669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4703.725075] LustreError: 104669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.836422] LustreError: 104670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.649801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.473764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4721.999175] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4736.832884] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4747.401144] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4760.742133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4761.674701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.776263] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4780.966870] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4784.178591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4799.457331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4799.471447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.483027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4799.512936] Lustre: Skipped 1 previous similar message [ 4801.029862] Lustre: server umount lustre-OST0000 complete [ 4809.697513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4809.701562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4809.702727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4809.702731] Lustre: Skipped 1 previous similar message [ 4809.720870] Lustre: Skipped 1 previous similar message [ 4814.660671] Lustre: server umount lustre-MDT0000 complete [ 4814.817723] LustreError: 105398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4814.869761] LustreError: 105398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4819.940510] LustreError: 104666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.952963] LustreError: 104666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4821.631699] LustreError: 104650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785393438 with bad export cookie 14291461420276929746 [ 4821.649126] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4822.695941] Lustre: server umount lustre-MDT0001 complete [ 4847.367635] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 4854.467840] Key type lgssc unregistered [ 4855.316212] LNet: 108253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4855.341612] LNetError: 108253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4855.430058] LNet: Removed LNI 192.168.202.111@tcp [ 4857.566953] Key type .llcrypt unregistered [ 4857.571981] Key type ._llcrypt unregistered [ 4888.253412] Key type ._llcrypt registered [ 4888.255358] Key type .llcrypt registered [ 4888.333297] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 4889.950810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4890.031021] alg: No test for adler32 (adler32-zlib) [ 4891.373699] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 4891.629532] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 4893.324104] Key type lgssc registered [ 4895.617860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4908.133072] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4908.167329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4923.745995] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4928.864526] LustreError: 109499:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 4955.552187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4964.753903] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 4975.801611] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4986.213116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.829890] LustreError: MGC192.168.202.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4986.841943] LustreError: 110528:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4986.846794] LustreError: 110528:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4986.851276] LustreError: 110528:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4986.938689] Lustre: server umount lustre-MDT0000 complete [ 4986.945619] LustreError: 110528:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5006.304223] Lustre: 108835:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785393606/real 1785393606] req@00000000eb542f0b x1872120783048064/t0(0) o400->MGC192.168.202.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785393622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5006.342261] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5006.364931] LustreError: 109511:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 5006.625434] Lustre: server umount lustre-OST0000 complete [ 5022.035859] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 5037.186497] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 5091.110460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.628479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5091.719663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5092.012064] Lustre: lustre-MDT0000: new disk, initializing [ 5092.175327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5092.196502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5102.415917] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5117.397805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.753983] Lustre: 113320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5117.923969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5117.939744] Lustre: Skipped 1 previous similar message [ 5118.271873] Lustre: lustre-MDT0001: new disk, initializing [ 5118.506994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5118.573204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5118.592496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5125.494688] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5141.190793] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5151.426772] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5160.414073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5160.897118] Lustre: lustre-OST0000: new disk, initializing [ 5160.903545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5160.909874] Lustre: 115293:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5161.065805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.222560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5168.241147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5168.435053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5170.423445] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5186.995671] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5195.840900] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5196.331266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5204.622962] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5204.957200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5209.569067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5209.579591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5209.592947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5209.619209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5209.670053] Lustre: Skipped 1 previous similar message [ 5214.693298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5214.727650] Lustre: Skipped 1 previous similar message [ 5219.811671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5222.368131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5222.535577] Lustre: server umount lustre-OST0000 complete [ 5230.052368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.053028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5230.076468] Lustre: Skipped 1 previous similar message [ 5230.094802] Lustre: Skipped 2 previous similar messages [ 5234.336508] Lustre: server umount lustre-MDT0000 complete [ 5239.343243] LustreError: 113314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785393855 with bad export cookie 15045269779578199727 [ 5239.355524] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5239.361170] LustreError: 113314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5240.059891] Lustre: server umount lustre-MDT0001 complete [ 5252.223645] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:44:25 (1785393865) [ 5257.274397] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 5284.610302] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 5302.476195] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 5358.462487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.949124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5358.998091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5359.126953] Lustre: lustre-MDT0000: new disk, initializing [ 5359.193088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5359.217389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5366.231558] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5381.070026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5381.389467] Lustre: 120227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5381.474751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5381.478184] Lustre: Skipped 1 previous similar message [ 5381.592843] Lustre: lustre-MDT0001: new disk, initializing [ 5381.684637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5381.715393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5381.724678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5390.013536] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5405.757538] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5414.394907] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5423.827902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5424.413451] Lustre: lustre-OST0000: new disk, initializing [ 5424.428671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5424.440844] Lustre: 122199:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5424.673055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5433.219939] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5434.461665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5434.483061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5434.670871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5447.148107] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5455.914320] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5456.534570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5465.721180] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5466.166492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5470.692459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5470.697851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5470.713753] LustreError: Skipped 1 previous similar message [ 5470.714684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5470.764485] Lustre: Skipped 1 previous similar message [ 5475.548484] Lustre: server umount lustre-OST0000 complete [ 5484.001529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5484.004581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.050987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5484.074614] Lustre: Skipped 1 previous similar message [ 5488.516202] Lustre: server umount lustre-MDT0000 complete [ 5491.171651] LustreError: 120235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.219479] LustreError: 120235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5496.304709] LustreError: 120238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.336400] LustreError: 120238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5496.703765] LustreError: 121023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785394113 with bad export cookie 15045269779578200609 [ 5496.726561] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5496.728544] LustreError: 121023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5498.225801] Lustre: server umount lustre-MDT0001 complete [ 5512.375737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5512.978429] LustreError: 124418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5513.089139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.311460] LustreError: 124419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.372979] LustreError: 124419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5520.438908] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5523.426602] LustreError: 124418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5532.223694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.965829] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5549.132852] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5557.732967] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5567.853750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5577.417296] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5588.282437] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5596.941053] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 5603.297941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.305773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.319018] Lustre: Skipped 1 previous similar message [ 5603.327333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.332724] Lustre: Skipped 2 previous similar messages [ 5607.369359] Lustre: server umount lustre-OST0000 complete [ 5614.561324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5614.563939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.566756] LustreError: Skipped 1 previous similar message [ 5614.589170] Lustre: Skipped 2 previous similar messages [ 5627.360147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5627.970217] Lustre: server umount lustre-MDT0000 complete [ 5629.935849] LustreError: 125148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.968153] LustreError: 125148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5634.350868] LustreError: 124399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785394250 with bad export cookie 15045269779578201491 [ 5634.360322] LustreError: 124399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5634.365185] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5635.174060] Lustre: server umount lustre-MDT0001 complete [ 5657.810782] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 5663.535870] Key type lgssc unregistered [ 5664.271397] LNet: 128076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5664.286458] LNetError: 128076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5664.334815] LNet: Removed LNI 192.168.202.111@tcp [ 5666.280727] Key type .llcrypt unregistered [ 5666.284885] Key type ._llcrypt unregistered [ 5695.291188] Key type ._llcrypt registered [ 5695.292448] Key type .llcrypt registered [ 5695.462485] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [ 5716.427668] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 5717.678099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5717.917436] alg: No test for adler32 (adler32-zlib) [ 5718.945982] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 5719.343737] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 5721.056138] Key type lgssc registered [ 5722.641621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5777.321855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5777.372210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.706321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5778.731457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5778.839756] Lustre: lustre-MDT0000: new disk, initializing [ 5778.963667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.001978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5786.514677] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5801.637819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.787373] Lustre: 131241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5801.866912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5801.875916] Lustre: Skipped 1 previous similar message [ 5802.012819] Lustre: lustre-MDT0001: new disk, initializing [ 5802.190284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5802.251507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5802.265867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5808.155457] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5823.952120] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5832.056798] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5841.299903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5841.924581] Lustre: lustre-OST0000: new disk, initializing [ 5841.929758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5841.940594] Lustre: 133214:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5842.049988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5849.753441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5849.768845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5849.903870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5851.636019] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5866.079027] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5874.247635] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5874.534426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5881.015678] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5881.492805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5885.935489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5885.949379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.961658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5885.965359] Lustre: Skipped 1 previous similar message [ 5885.989601] Lustre: Skipped 1 previous similar message [ 5890.857426] Lustre: server umount lustre-OST0000 complete [ 5899.751093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5899.756512] LustreError: Skipped 1 previous similar message [ 5899.764640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.790176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5901.286171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.299479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5901.333987] Lustre: Skipped 1 previous similar message [ 5903.050314] Lustre: server umount lustre-MDT0000 complete [ 5909.202806] LustreError: 131235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785394525 with bad export cookie 6148308071023491097 [ 5909.218648] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5909.218753] LustreError: 131235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5910.037546] Lustre: server umount lustre-MDT0001 complete [ 5922.860932] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:55:36 (1785394536) [ 5932.450692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5933.274427] LustreError: 135632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.285215] LustreError: 135632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5933.374744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5938.660557] LustreError: 135633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.815222] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5942.819582] LustreError: 135632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5947.873642] LustreError: 135633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.805263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.993641] LustreError: 135632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5953.419514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5959.734682] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 5971.367653] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5979.047822] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5983.201424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.223302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5983.230759] Lustre: Skipped 1 previous similar message [ 5988.134561] Lustre: server umount lustre-MDT0000 complete [ 5988.327513] LustreError: 136360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.346141] LustreError: 136360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5992.917531] LustreError: 135612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785394609 with bad export cookie 6148308071023491979 [ 5992.923465] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5992.928817] LustreError: 135612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5993.466642] Lustre: server umount lustre-MDT0001 complete [ 6004.493196] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:56:58 (1785394618) [ 6013.113451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6028.640564] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6033.760471] LustreError: 138048:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 6059.488288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6067.286375] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6077.332496] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6080.076396] Lustre: server umount lustre-OST0000 complete [ 6090.366626] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:58:24 (1785394704) [ 6099.002125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.573613] LustreError: 139495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.669033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.505535] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6118.753921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.224406] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6137.087158] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6144.730095] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6154.962341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6155.452171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6155.461762] Lustre: Skipped 1 previous similar message [ 6164.609236] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6175.812719] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6191.073435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6191.086270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.096874] Lustre: Skipped 1 previous similar message [ 6191.104968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6193.511135] Lustre: server umount lustre-MDT0000 complete [ 6196.720385] LustreError: 139495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.749946] LustreError: 139495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6198.763686] LustreError: 139477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785394815 with bad export cookie 6148308071023492644 [ 6198.783255] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6199.686901] Lustre: server umount lustre-MDT0001 complete [ 6215.864041] Lustre: server umount lustre-OST0000 complete [ 6227.169122] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 03:00:41 (1785394841) [ 6236.565551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.445068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.892288] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6256.722736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.936855] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6274.550983] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6283.284889] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6293.505032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6307.490363] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6307.779314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 6319.426247] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6326.588701] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6326.746913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6333.460750] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6333.811909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6338.529389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6338.529989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6338.544298] LustreError: Skipped 1 previous similar message [ 6338.544931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6338.544934] Lustre: Skipped 2 previous similar messages [ 6338.588540] Lustre: Skipped 3 previous similar messages [ 6342.677856] Lustre: server umount lustre-OST0000 complete [ 6349.298995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6355.953021] Lustre: server umount lustre-MDT0000 complete [ 6359.009581] LustreError: 143145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6359.018226] LustreError: 143145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6363.339500] LustreError: 145010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785394979 with bad export cookie 6148308071023493715 [ 6363.371042] LustreError: 145010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6363.385146] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6365.225484] Lustre: server umount lustre-MDT0001 complete [ 6379.217846] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 03:03:12 (1785394992) [ 6388.972489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.704771] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6409.824956] LustreError: 147328:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 6435.618189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.624254] Lustre: Skipped 2 previous similar messages [ 6444.634717] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6455.832124] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6465.006266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.846698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6473.021406] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6488.514644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.299860] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6507.709724] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6514.714465] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6521.578584] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6521.922330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6529.684458] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6530.094836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6535.144737] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.147587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6535.156816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6535.156827] Lustre: Skipped 4 previous similar messages [ 6535.172820] Lustre: Skipped 4 previous similar messages [ 6539.650194] Lustre: server umount lustre-MDT0000 complete [ 6540.265644] LustreError: 148976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.298341] LustreError: 148976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6546.125173] LustreError: 147334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395162 with bad export cookie 6148308071023494737 [ 6546.148176] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6547.576741] Lustre: server umount lustre-MDT0001 complete [ 6565.693141] Lustre: server umount lustre-OST0000 complete [ 6577.934902] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 03:06:31 (1785395191) [ 6587.736984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6603.425324] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6608.544417] LustreError: 151531:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 6644.781638] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6658.932820] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6674.125022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.049334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6681.765910] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6693.515169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.298707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6694.312359] Lustre: Skipped 4 previous similar messages [ 6702.019126] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6713.084441] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6722.587412] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6735.359908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6735.747964] Lustre: lustre-OST0001: new disk, initializing [ 6735.755204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6735.764723] Lustre: 154410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6745.780900] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6746.262456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6746.276361] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6746.489290] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6759.586296] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6767.510777] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6767.922336] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6775.729737] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6776.080256] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6781.410476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6781.419326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.434182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6781.444964] Lustre: Skipped 2 previous similar messages [ 6794.209077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6794.424179] Lustre: server umount lustre-OST0000 complete [ 6797.796363] LustreError: 153504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.841336] LustreError: 153504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6806.468262] Lustre: server umount lustre-OST0001 complete [ 6818.947673] Lustre: server umount lustre-MDT0000 complete [ 6824.337984] LustreError: 153311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395440 with bad export cookie 6148308071023495689 [ 6824.370689] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6824.376725] LustreError: 153311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6824.947859] Lustre: server umount lustre-MDT0001 complete [ 6856.759437] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 03:11:10 (1785395470) [ 6859.131383] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 6861.865312] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 6865.049462] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 03:11:18 (1785395478) [ 6874.574242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.076697] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6875.133645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6882.962209] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6894.749689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.924530] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6895.010628] Lustre: 157976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6901.682756] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6911.432470] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6918.444715] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6926.608148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.938908] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6935.877514] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 6935.926250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6945.888062] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6953.439197] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6953.852682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6960.436962] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6960.755149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6966.759586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.762216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6966.770058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6966.770066] Lustre: Skipped 12 previous similar messages [ 6966.775229] Lustre: Skipped 6 previous similar messages [ 6966.786448] LustreError: Skipped 3 previous similar messages [ 6969.259976] Lustre: server umount lustre-OST0000 complete [ 6999.661251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.518887] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7006.534162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7006.544800] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7006.590587] Lustre: Skipped 1 previous similar message [ 7010.515086] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7022.306468] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7031.490853] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7031.726321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7042.417553] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7042.826534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7051.887987] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a7185202000.ost_server_uuid 50 [ 7054.555788] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a7185202000.ost_server_uuid in IDLE state after 0 sec [ 7054.887343] LustreError: 157983:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.11@tcp) returned error from blocking AST (req@000000004ebdc353 x1872121650536192 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000005d943c8b/0x55532d9d8f05d363 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.11@tcp remote: 0x4e40f17742a562b7 expref: 5 pid: 157985 timeout: 7154 lvb_type: 0 lru_score: 0 lru_type: 0 [ 7054.912455] LustreError: lustre-MDT0000: A client on nid 192.168.202.11@tcp was evicted due to a lock blocking callback time out: rc -107 [ 7054.926652] LustreError: 157973:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.11@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000005d943c8b/0x55532d9d8f05d363 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.11@tcp remote: 0x4e40f17742a562b7 expref: 6 pid: 157985 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 7063.016908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7063.033618] LustreError: Skipped 2 previous similar messages [ 7068.078802] Lustre: server umount lustre-OST0000 complete [ 7079.795118] Lustre: server umount lustre-MDT0000 complete [ 7085.717307] LustreError: 157968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395702 with bad export cookie 6148308071023496872 [ 7085.718461] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7085.732331] LustreError: 157968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7100.384443] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7101.187503] Lustre: server umount lustre-MDT0001 complete [ 7121.591405] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7126.557803] Key type lgssc unregistered [ 7127.027704] LNet: 164128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7127.050311] LNetError: 164128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7127.093662] LNet: Removed LNI 192.168.202.111@tcp [ 7128.335412] Key type .llcrypt unregistered [ 7128.340533] Key type ._llcrypt unregistered [ 7165.907633] Key type ._llcrypt registered [ 7165.911449] Key type .llcrypt registered [ 7166.096350] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7167.623033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7167.651815] alg: No test for adler32 (adler32-zlib) [ 7169.075132] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 7169.493439] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7171.352193] Key type lgssc registered [ 7173.541964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7185.739757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7185.756900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7187.435670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7193.434712] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7196.667346] LustreError: 165559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.697049] LustreError: 165559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7201.768864] LustreError: 165560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.886235] LustreError: 165559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.809191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.422641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7216.980395] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7229.738915] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7237.714453] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7248.443410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.496673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.719824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7262.433124] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7271.595162] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7279.098865] Lustre: Failing over lustre-MDT0000 [ 7279.417229] LustreError: 165556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.806896] Lustre: server umount lustre-MDT0000 complete [ 7280.113325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7280.127986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.674093] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.795088] LustreError: 165560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.822728] LustreError: 165560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7292.669324] LustreError: 165538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.11@tcp arrived at 1785395909 with bad export cookie 11019068572856870136 [ 7292.707572] LustreError: 165538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7298.027909] LustreError: 165560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7298.049032] LustreError: 165560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7299.045209] Lustre: 164868:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785395899/real 1785395899] req@00000000d352e521 x1872123171735296/t0(0) o400->MGC192.168.202.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785395915 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7299.066811] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.189550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.576080] LustreError: 168614:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7308.579945] LustreError: 168614:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000000be68ae x1872123171739136/t0(0) o250->MGC192.168.202.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1785395925 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7308.600093] LustreError: 168614:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7309.316554] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x98eb987c48cde55a [ 7309.334406] Lustre: MGC192.168.202.111@tcp: Connection restored to 0@lo (at 0@lo) [ 7309.533023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7309.568124] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7314.921915] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7314.947195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7317.237179] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7320.561867] Lustre: lustre-MDT0000: Denying connection for new client 31cc38c7-526c-4790-bb45-ff17ff93b6e5 (at 192.168.202.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 7326.000712] Lustre: lustre-MDT0000: Denying connection for new client 31cc38c7-526c-4790-bb45-ff17ff93b6e5 (at 192.168.202.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 7334.884036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7334.886594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.918372] Lustre: Skipped 1 previous similar message [ 7334.926666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7337.853611] Lustre: server umount lustre-OST0000 complete [ 7344.112270] LustreError: 169298:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 7344.135537] Lustre: 168646:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 7344.159077] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 7344.187903] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 7344.206381] LustreError: 168646:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bb9bcf5d x1872123171756928/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7344.236743] LustreError: 168646:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7344.258687] LustreError: 168646:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 7344.302593] Lustre: lustre-MDT0000: Recovery over after 0:30, of 2 clients 0 recovered and 2 were evicted. [ 7345.421667] Lustre: server umount lustre-MDT0000 complete [ 7345.639844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7345.666081] Lustre: Skipped 1 previous similar message [ 7345.686074] LustreError: 165556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.743717] LustreError: 165556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7352.284469] LustreError: 168143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785395968 with bad export cookie 11019068572856870234 [ 7352.291547] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.299154] LustreError: 168143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7352.329335] LustreError: 164867:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 7359.737271] Lustre: server umount lustre-MDT0001 complete [ 7381.318759] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7387.450760] Key type lgssc unregistered [ 7388.132083] LNet: 170074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7388.139934] LNetError: 170074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7388.163771] LNet: Removed LNI 192.168.202.111@tcp [ 7390.016461] Key type .llcrypt unregistered [ 7390.018639] Key type ._llcrypt unregistered [ 7424.145234] Key type ._llcrypt registered [ 7424.147844] Key type .llcrypt registered [ 7424.357904] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7426.307631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7426.571405] alg: No test for adler32 (adler32-zlib) [ 7427.866584] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 7428.204064] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7429.936175] Key type lgssc registered [ 7431.080855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7441.697426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7441.747423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.552173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.801405] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7453.666418] LustreError: 171490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.683333] LustreError: 171490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7458.785914] LustreError: 171489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7463.907904] LustreError: 171490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.771448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.418747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.213690] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7485.374158] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7496.486713] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7508.506467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7509.394654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7513.509465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7522.260484] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7533.995441] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.289732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7544.298974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7544.302076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7544.334556] Lustre: Skipped 1 previous similar message [ 7548.840040] Lustre: server umount lustre-OST0000 complete [ 7554.528961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7554.531213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7554.556059] Lustre: Skipped 1 previous similar message [ 7554.560071] Lustre: Skipped 1 previous similar message [ 7559.662754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7559.671678] Lustre: Skipped 2 previous similar messages [ 7560.904671] Lustre: server umount lustre-MDT0000 complete [ 7564.784363] LustreError: 171485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.826403] LustreError: 171485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7567.413833] LustreError: 171471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396183 with bad export cookie 2019208638415164749 [ 7567.440773] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7567.458343] LustreError: 171471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7568.759972] Lustre: server umount lustre-MDT0001 complete [ 7588.163632] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7592.919906] Key type lgssc unregistered [ 7593.553381] LNet: 175053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7593.570759] LNetError: 175053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7593.628594] LNet: Removed LNI 192.168.202.111@tcp [ 7595.385143] Key type .llcrypt unregistered [ 7595.390277] Key type ._llcrypt unregistered [ 7649.469256] Key type ._llcrypt registered [ 7649.473239] Key type .llcrypt registered [ 7649.706729] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7650.961237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7651.022096] alg: No test for adler32 (adler32-zlib) [ 7652.578618] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 7652.857310] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7654.666066] Key type lgssc registered [ 7656.370494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7667.948200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7667.999168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.160622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.704195] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7679.463526] LustreError: 176760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.488442] LustreError: 176760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7684.590251] LustreError: 176761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7687.726967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.207047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7694.708134] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7705.040892] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7714.002823] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7723.924280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7724.686596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7735.000598] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7741.133789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 7747.867013] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7758.817039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7758.838898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.874987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7761.890193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7761.894100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7761.909375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.951718] Lustre: Skipped 1 previous similar message [ 7764.361562] Lustre: server umount lustre-OST0000 complete [ 7770.081483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7770.097457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.127867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7772.133739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7772.134146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.095705] Lustre: server umount lustre-MDT0000 complete [ 7777.253368] LustreError: 176761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7777.285516] LustreError: 176761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7781.993393] LustreError: 178607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396398 with bad export cookie 9991651998761874320 [ 7782.011153] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7782.014660] LustreError: 178607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7782.703513] Lustre: server umount lustre-MDT0001 complete [ 7802.108938] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7808.045774] Key type lgssc unregistered [ 7808.894497] LNet: 180300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7808.905105] LNetError: 180300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7808.948063] LNet: Removed LNI 192.168.202.111@tcp [ 7811.362461] Key type .llcrypt unregistered [ 7811.368203] Key type ._llcrypt unregistered [ 7845.070154] Key type ._llcrypt registered [ 7845.083534] Key type .llcrypt registered [ 7845.226868] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 7846.706038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7846.876242] alg: No test for adler32 (adler32-zlib) [ 7848.084878] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 7848.639848] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 7850.562967] Key type lgssc registered [ 7852.580776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7868.615933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7868.699995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7870.352838] Lustre: *** cfs_fail_loc=135, val=0*** [ 7870.600857] LustreError: 181826:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 7870.608946] LustreError: 181826:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.111@tcp: cfg command failed: rc = -2 [ 7870.622382] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7870.622382] [ 7870.638285] LustreError: MGC192.168.202.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7870.657916] LustreError: 181787:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7870.666787] LustreError: 181787:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7870.676621] LustreError: 181787:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7870.688056] LustreError: 181787:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7870.865635] Lustre: server umount lustre-MDT0000 complete [ 7870.879708] LustreError: 181787:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7880.665031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7881.896647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7888.213694] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7891.435053] LustreError: 182197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.452768] LustreError: 182197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7896.594793] LustreError: 182198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.678274] LustreError: 182197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.884186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.922754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7913.321356] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 7926.974802] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7939.578078] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7944.427720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7944.446113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7949.616866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7949.622576] Lustre: Skipped 2 previous similar messages [ 7951.464829] Lustre: server umount lustre-MDT0000 complete [ 7954.930531] LustreError: 183003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7954.941881] LustreError: 183003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7957.061124] LustreError: 182179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396573 with bad export cookie 14621933806719090920 [ 7957.066112] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7957.074338] LustreError: 182179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7958.204698] Lustre: server umount lustre-MDT0001 complete [ 7975.838122] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 7981.505819] Key type lgssc unregistered [ 7982.034634] LNet: 184528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7982.050880] LNetError: 184528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7983.078715] LNet: Removed LNI 192.168.202.111@tcp [ 7984.625483] Key type .llcrypt unregistered [ 7984.629576] Key type ._llcrypt unregistered [ 8015.516140] Key type ._llcrypt registered [ 8015.520701] Key type .llcrypt registered [ 8015.738220] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 8016.262399] Key type .llcrypt unregistered [ 8016.268228] Key type ._llcrypt unregistered [ 8039.749662] Key type ._llcrypt registered [ 8039.754150] Key type .llcrypt registered [ 8039.963613] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8041.627364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8041.959480] alg: No test for adler32 (adler32-zlib) [ 8043.322298] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 8043.818672] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 8045.584256] Key type lgssc registered [ 8048.102239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8058.542979] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8058.591377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8074.400527] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8079.520433] LustreError: 186677:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.111@tcp: failed processing log, type 4: rc = -110 [ 8106.336255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.250828] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8127.492290] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8136.658571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.267411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.316725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 8144.087401] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8146.418509] LustreError: 186703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.440174] LustreError: 186703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8151.522378] LustreError: 186702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8151.550314] LustreError: 186702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8156.660246] LustreError: 186703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8156.717930] LustreError: 186703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8157.571864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8158.440881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8165.171734] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8175.348614] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8185.748501] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8190.957560] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8208.872305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8208.888440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.899322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8209.902463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8209.922987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8209.931280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.987879] Lustre: Skipped 1 previous similar message [ 8212.899830] Lustre: server umount lustre-OST0000 complete [ 8224.751525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8224.773425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8224.805717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8227.177756] Lustre: server umount lustre-MDT0000 complete [ 8230.376582] LustreError: 187716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8230.403179] LustreError: 187716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8233.758956] LustreError: 186685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396850 with bad export cookie 10562482798642817783 [ 8233.766346] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8233.770192] LustreError: 186685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8234.730840] Lustre: server umount lustre-MDT0001 complete [ 8256.640511] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 8264.691510] Key type lgssc unregistered [ 8265.583537] LNet: 190619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8265.598748] LNetError: 190619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8265.650943] LNet: Removed LNI 192.168.202.111@tcp [ 8268.507379] Key type .llcrypt unregistered [ 8268.509073] Key type ._llcrypt unregistered [ 8313.886188] Key type ._llcrypt registered [ 8313.895835] Key type .llcrypt registered [ 8314.105928] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8315.688531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8315.951616] alg: No test for adler32 (adler32-zlib) [ 8317.324850] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 8317.815142] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 8319.576192] Key type lgssc registered [ 8321.493628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8332.160207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8332.173533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8333.667701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.804535] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8344.035803] LustreError: 192037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8344.050657] LustreError: 192037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8349.164373] LustreError: 192038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8351.492455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8352.399871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8358.214405] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8367.319270] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8375.195378] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8385.125127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8385.845939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8399.449855] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8404.336251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 8415.534329] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8424.788442] Lustre: Failing over lustre-MDT0000 [ 8424.932939] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.949560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8425.458622] Lustre: server umount lustre-MDT0000 complete [ 8428.873092] LustreError: 192034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.966571] LustreError: 192032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8434.001602] LustreError: 192032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8439.121750] LustreError: 193885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.185453] LustreError: 193885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8446.432500] Lustre: 191349:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785397046/real 1785397046] req@00000000c947ce3f x1872124375501440/t0(0) o400->MGC192.168.202.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785397062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8446.462567] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8447.398291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.327167] LustreError: 194045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8449.347527] LustreError: 194045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8457.279264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8457.413827] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8459.570937] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8462.329380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8462.378524] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8462.488837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 8466.944415] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8475.534287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 8487.238951] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 8499.691295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8499.706553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.735731] Lustre: Skipped 2 previous similar messages [ 8499.757572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8499.769293] Lustre: Skipped 2 previous similar messages [ 8503.267995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8503.278878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.274413] Lustre: server umount lustre-OST0000 complete [ 8513.507457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8513.512845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8513.540150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8513.540157] Lustre: Skipped 1 previous similar message [ 8516.556324] Lustre: server umount lustre-MDT0000 complete [ 8518.626839] LustreError: 192038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8518.647424] LustreError: 192038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 8522.148765] LustreError: 192016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397138 with bad export cookie 7991893859897282866 [ 8522.157867] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.159441] LustreError: 192016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8523.033219] Lustre: server umount lustre-MDT0001 complete [ 8538.556724] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 8542.734284] Key type lgssc unregistered [ 8543.184598] LNet: 196940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8543.187711] LNetError: 196940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8543.208684] LNet: Removed LNI 192.168.202.111@tcp [ 8544.794143] Key type .llcrypt unregistered [ 8544.798948] Key type ._llcrypt unregistered [ 8575.596409] Key type ._llcrypt registered [ 8575.602629] Key type .llcrypt registered [ 8575.758785] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8577.266720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8577.333025] alg: No test for adler32 (adler32-zlib) [ 8578.682110] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 8578.942320] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 8580.824431] Key type lgssc registered [ 8582.699531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8592.393557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8592.442175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.128603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.613512] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8603.632497] LustreError: 198355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8603.657752] LustreError: 198355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8608.739235] LustreError: 198356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.867765] LustreError: 198355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.598792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8615.693567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8624.402343] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8638.545232] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8646.192741] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8655.987710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8656.388289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8663.590069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 8664.108911] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8677.107634] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8682.830470] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8689.273173] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 8709.018393] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8724.964137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8724.981107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8725.020586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8725.594178] Lustre: server umount lustre-OST0000 complete [ 8733.155749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8733.159383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8733.186383] Lustre: Skipped 1 previous similar message [ 8738.278874] LustreError: 198350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8738.295546] LustreError: 198350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8740.115419] Lustre: server umount lustre-MDT0000 complete [ 8743.342945] Lustre: 197665:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785397343/real 1785397343] req@00000000af219e95 x1872124649201792/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785397359 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8743.392753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8743.393772] LustreError: 198356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8743.433210] LustreError: 198356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8744.416371] Lustre: 197667:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785397344/real 1785397344] req@00000000d4b76730 x1872124649202560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785397360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8745.459800] LustreError: 198337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397362 with bad export cookie 16224171122072387017 [ 8745.467759] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8745.479176] LustreError: 198337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8746.001535] Lustre: server umount lustre-MDT0001 complete [ 8761.794150] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 8765.727576] Key type lgssc unregistered [ 8766.114664] LNet: 202241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8766.118747] LNetError: 202241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8766.144871] LNet: Removed LNI 192.168.202.111@tcp [ 8767.109486] Key type .llcrypt unregistered [ 8767.112636] Key type ._llcrypt unregistered [ 8795.006274] Key type ._llcrypt registered [ 8795.007813] Key type .llcrypt registered [ 8795.197928] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 8796.697626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8796.882542] alg: No test for adler32 (adler32-zlib) [ 8798.417919] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 8798.962370] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 8800.845275] Key type lgssc registered [ 8803.128199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8814.856313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8814.925967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.801476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8823.955433] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8827.361478] LustreError: 203511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8827.393254] LustreError: 203511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8832.498252] LustreError: 203512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8836.575519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8836.823553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8843.214743] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8856.563955] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8865.223950] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8875.197258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8875.627141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8884.905740] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 8887.924463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8898.086508] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8905.533391] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 8918.225934] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8932.195787] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 8948.196723] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 8970.214797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8970.223333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.244146] LustreError: Skipped 1 previous similar message [ 8970.278313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8975.442914] Lustre: server umount lustre-OST0000 complete [ 8985.569367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8985.588641] Lustre: Skipped 1 previous similar message [ 8985.610175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8985.617434] Lustre: Skipped 2 previous similar messages [ 8989.739047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8989.755511] Lustre: Skipped 1 previous similar message [ 8994.785678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8994.796046] Lustre: Skipped 1 previous similar message [ 8995.298042] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8995.672372] Lustre: server umount lustre-MDT0000 complete [ 8999.908370] LustreError: 204446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.942460] LustreError: 204446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9001.693604] LustreError: 206940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397618 with bad export cookie 5342391773934196108 [ 9001.701615] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9001.711321] LustreError: 206940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9002.598627] Lustre: server umount lustre-MDT0001 complete [ 9020.083941] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9024.230993] Key type lgssc unregistered [ 9024.899306] LNet: 208541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9024.922054] LNetError: 208541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9024.964097] LNet: Removed LNI 192.168.202.111@tcp [ 9025.868516] Key type .llcrypt unregistered [ 9025.873152] Key type ._llcrypt unregistered [ 9063.501555] Key type ._llcrypt registered [ 9063.507708] Key type .llcrypt registered [ 9063.844666] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9066.621656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9066.789854] alg: No test for adler32 (adler32-zlib) [ 9068.779985] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 9069.158430] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9071.217885] Key type lgssc registered [ 9074.187295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9091.390438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9091.453886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.315502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9100.329726] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9102.852438] LustreError: 209957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9102.898070] LustreError: 209957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9107.943098] LustreError: 209958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9111.607549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.271107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.051598] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9130.186711] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9139.169608] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9148.706051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9149.412076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.417906] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9163.773815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 9169.432663] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9192.427304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9192.437758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.453131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9194.468360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9194.481384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.001523] Lustre: server umount lustre-OST0000 complete [ 9204.198580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9204.217603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.252154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9204.259904] Lustre: Skipped 1 previous similar message [ 9209.836161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9209.851849] Lustre: Skipped 2 previous similar messages [ 9210.557715] Lustre: server umount lustre-MDT0000 complete [ 9214.945516] LustreError: 212455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.962153] LustreError: 212455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9216.475378] LustreError: 209937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397833 with bad export cookie 1512434546302154363 [ 9216.477733] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9216.493215] LustreError: 209937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9217.177556] Lustre: server umount lustre-MDT0001 complete [ 9233.706854] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9237.647910] Key type lgssc unregistered [ 9237.999900] LNet: 213649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9238.008728] LNetError: 213649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9238.025641] LNet: Removed LNI 192.168.202.111@tcp [ 9239.017257] Key type .llcrypt unregistered [ 9239.018867] Key type ._llcrypt unregistered [ 9270.481647] Key type ._llcrypt registered [ 9270.483492] Key type .llcrypt registered [ 9270.575675] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9272.380595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9272.421289] alg: No test for adler32 (adler32-zlib) [ 9273.388743] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 9273.751905] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9275.625177] Key type lgssc registered [ 9277.893536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9290.200277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9290.227325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.945866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9298.925879] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9301.482730] LustreError: 214917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9301.513343] LustreError: 214917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9306.605840] LustreError: 214918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9311.156840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.636159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9319.377538] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9332.880541] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9343.063774] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9356.925404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9358.115460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.171104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 9370.452392] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9382.986599] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9411.806368] Lustre: server umount lustre-OST0000 complete [ 9419.233531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9419.239884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9419.265643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9422.819552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9422.823917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.945083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9427.962030] Lustre: Skipped 2 previous similar messages [ 9428.960205] Lustre: 214229:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785398029/real 1785398029] req@00000000dc1d5734 x1872125377944064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785398045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9429.017513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.041251] LustreError: 215976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9429.078379] LustreError: 215976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9429.392163] Lustre: server umount lustre-MDT0000 complete [ 9434.082837] Lustre: 214228:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785398034/real 1785398034] req@000000000aea5061 x1872125377945472/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785398050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9434.087741] LustreError: 214918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.121272] Lustre: 214228:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9434.169244] LustreError: 214918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9436.191992] LustreError: 214898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398052 with bad export cookie 15067790494479465510 [ 9436.211050] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9436.213184] LustreError: 214898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9436.983589] Lustre: server umount lustre-MDT0001 complete [ 9456.859651] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9460.800226] Key type lgssc unregistered [ 9461.316731] LNet: 218653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9461.354095] LNetError: 218653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9461.402233] LNet: Removed LNI 192.168.202.111@tcp [ 9463.067486] Key type .llcrypt unregistered [ 9463.070458] Key type ._llcrypt unregistered [ 9487.213331] Key type ._llcrypt registered [ 9487.218611] Key type .llcrypt registered [ 9487.390707] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9488.934910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9489.033923] alg: No test for adler32 (adler32-zlib) [ 9490.254428] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 9490.533938] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9492.216577] Key type lgssc registered [ 9493.488618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9504.578191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9504.609515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9506.263342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9515.490064] LustreError: 219928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9515.518321] LustreError: 219928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9515.525437] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9520.610071] LustreError: 219929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9525.746223] LustreError: 219928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9527.797652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9528.635257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9535.127773] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9547.007700] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9556.159279] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9565.551766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9566.037084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9573.541312] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9576.300870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 9585.068157] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9598.424789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9598.608564] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9599.092436] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9610.081854] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9619.929186] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9622.244594] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9635.738430] Lustre: Permanently deactivating lustre-OST0001 [ 9635.749867] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9640.987988] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 9657.446187] Lustre: Permanently reactivating lustre-OST0001 [ 9657.459737] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 9657.464564] Lustre: Skipped 2 previous similar messages [ 9666.145841] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9666.157687] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 9666.172292] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9666.190964] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9666.245096] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9671.651493] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9671.651897] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9671.663425] LustreError: Skipped 1 previous similar message [ 9671.665785] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9671.673071] Lustre: Skipped 2 previous similar messages [ 9671.691392] Lustre: Skipped 1 previous similar message [ 9673.188245] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9675.962413] Lustre: server umount lustre-OST0001 complete [ 9678.311180] LustreError: 221753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.344145] LustreError: 221753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9681.893826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9681.901511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9681.917990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9681.936127] Lustre: Skipped 1 previous similar message [ 9683.429035] LustreError: 221751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.471701] LustreError: 221751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9688.006047] Lustre: server umount lustre-OST0000 complete [ 9697.761684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9697.768513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9697.802401] Lustre: Skipped 1 previous similar message [ 9697.815946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9697.825304] Lustre: Skipped 2 previous similar messages [ 9700.584788] Lustre: server umount lustre-MDT0000 complete [ 9703.917976] LustreError: 219929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.940551] LustreError: 219929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9705.639190] LustreError: 221776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398322 with bad export cookie 4188262256454303065 [ 9705.658691] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9705.663040] LustreError: 221776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9706.460605] Lustre: server umount lustre-MDT0001 complete [ 9722.170098] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [ 9725.340717] Key type lgssc unregistered [ 9725.686553] LNet: 224997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9725.692464] LNetError: 224997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9725.711894] LNet: Removed LNI 192.168.202.111@tcp [ 9726.839332] Key type .llcrypt unregistered [ 9726.840478] Key type ._llcrypt unregistered [ 9782.114539] Key type ._llcrypt registered [ 9782.118192] Key type .llcrypt registered [ 9782.297649] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [ 9784.522436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9784.666257] alg: No test for adler32 (adler32-zlib) [ 9785.964331] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [ 9786.448745] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [ 9788.298323] Key type lgssc registered [ 9790.056556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9801.905266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9801.920399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.344996] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9803.383849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9803.744148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9811.333646] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9822.497673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.671186] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9822.751536] Lustre: 227209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9823.324489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9828.778907] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9839.662605] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9847.949857] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9858.674201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9859.037977] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9859.446483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.541524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 9869.905545] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [ 9880.664792] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9884.364596] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9892.357642] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 9900.037360] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 9906.122198] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9921.962221] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9921.972799] Lustre: Skipped 1 previous similar message [ 9944.399396] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9944.411923] Lustre: Skipped 2 previous similar messages [ 9985.957037] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9985.964716] Lustre: Skipped 4 previous similar messages [10055.365603] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [10055.369677] Lustre: Skipped 7 previous similar messages [10067.958539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10067.985045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.000735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10073.058639] Lustre: server umount lustre-OST0000 complete [10083.817159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10083.833668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10083.842139] Lustre: Skipped 2 previous similar messages [10083.871510] Lustre: Skipped 3 previous similar messages [10087.417100] Lustre: server umount lustre-MDT0000 complete [10092.576154] LustreError: 227200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398709 with bad export cookie 6874571541851624019 [10092.576731] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10092.590422] LustreError: 227200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10093.146509] Lustre: server umount lustre-MDT0001 complete [10113.829950] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [10119.248836] Key type lgssc unregistered [10119.801712] LNet: 232046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10119.815228] LNetError: 232046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10119.855679] LNet: Removed LNI 192.168.202.111@tcp [10121.610949] Key type .llcrypt unregistered [10121.615891] Key type ._llcrypt unregistered [10161.674092] Key type ._llcrypt registered [10161.688714] Key type .llcrypt registered [10161.892658] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10164.833913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10164.988803] alg: No test for adler32 (adler32-zlib) [10166.469255] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [10167.164441] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [10169.048290] Key type lgssc registered [10172.427883] Lustre: Echo OBD driver; http://www.lustre.org/ [10184.807200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10184.855446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10186.581834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10192.909204] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10196.963856] LustreError: 233463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.984498] LustreError: 233463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10202.092776] LustreError: 233464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10204.936403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10205.426606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10211.022262] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10222.777348] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10230.221271] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10240.050700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10240.704870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10250.026679] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_default_debug -1 all [10257.131150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [10260.780231] Lustre: DEBUG MARKER: oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10282.977930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10282.987783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10282.999287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10283.015902] Lustre: Skipped 1 previous similar message [10283.036929] Lustre: Skipped 1 previous similar message [10288.097820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10288.112983] Lustre: Skipped 1 previous similar message [10293.230599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10293.246256] Lustre: Skipped 1 previous similar message [10296.288323] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10296.897725] Lustre: server umount lustre-OST0000 complete [10302.437469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10302.443226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10302.479090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10303.462277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10308.598380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10308.610578] Lustre: Skipped 3 previous similar messages [10308.870268] Lustre: server umount lustre-MDT0000 complete [10313.699218] LustreError: 233463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.718613] LustreError: 233463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10314.132669] LustreError: 235309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398930 with bad export cookie 14477995569214548234 [10314.168318] LustreError: MGC192.168.202.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10314.912593] Lustre: server umount lustre-MDT0001 complete [10331.152500] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [10334.890539] Key type lgssc unregistered [10335.251585] LNet: 237161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10335.264955] LNetError: 237161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10335.283819] LNet: Removed LNI 192.168.202.111@tcp [10336.756084] Key type .llcrypt unregistered [10336.759664] Key type ._llcrypt unregistered [10372.053714] Key type ._llcrypt registered [10372.058975] Key type .llcrypt registered [10372.272283] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10373.521486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10373.680726] alg: No test for adler32 (adler32-zlib) [10374.808746] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [10375.054119] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [10376.752162] Key type lgssc registered [10378.434838] Lustre: Echo OBD driver; http://www.lustre.org/ [10654.349472] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [10657.908380] Key type lgssc unregistered [10658.190158] LNet: 239184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10658.215169] LNetError: 239184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10658.234597] LNet: Removed LNI 192.168.202.111@tcp [10659.355191] Key type .llcrypt unregistered [10659.359876] Key type ._llcrypt unregistered [10698.986238] Key type ._llcrypt registered [10698.989359] Key type .llcrypt registered [10699.224821] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing set_hostid [10723.704812] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing load_modules_local [10725.481815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10725.744495] alg: No test for adler32 (adler32-zlib) [10726.869410] Lustre: Lustre: Build Version: 2.17.56_2_gcea94ac [10727.323153] LNet: Added LNI 192.168.202.111@tcp [8/256/0/180] [10729.065229] Key type lgssc registered [10731.267707] Lustre: Echo OBD driver; http://www.lustre.org/ [10790.819480] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10355 sec ========= 04:16:43 (1785399403) [10794.094960] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:16:47 (1785399407) === [10799.761938] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:16:53 (1785399413) === [10860.711228] Lustre: DEBUG MARKER: oleg211-server.virtnet: executing unload_modules_local [10865.760363] Key type lgssc unregistered [10866.257548] LNet: 244437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10866.273354] LNetError: 244437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10866.306114] LNet: Removed LNI 192.168.202.111@tcp [10867.594469] Key type .llcrypt unregistered [10867.597599] Key type ._llcrypt unregistered