[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 590899237 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003057] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010144] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013081] *** VALIDATE selinux *** [ 0.021828] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025842] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] *** VALIDATE tmpfs *** [ 0.030425] *** VALIDATE proc *** [ 0.031286] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040086] debug: unmapping init [mem 0xffffffffaca59000-0xffffffffaca60fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043680] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045025] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.052217] rcu: Hierarchical SRCU implementation. [ 0.054428] smp: Bringing up secondary CPUs ... [ 0.055597] x86: Booting SMP configuration: [ 0.056030] .... node #0, CPUs: #1 #2 #3 [ 0.059630] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228257] node 0 deferred pages initialised in 163ms [ 0.231139] devtmpfs: initialized [ 0.233186] x86/mm: Memory block size: 128MB [ 0.235855] gcov: version magic: 0x41383552 [ 0.236612] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.241026] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243209] pinctrl core: initialized pinctrl subsystem [ 0.244181] [ 0.244825] ************************************************************* [ 0.245022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246018] ** ** [ 0.247020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248019] ** ** [ 0.249017] ** This means that this kernel is built to expose internal ** [ 0.250018] ** IOMMU data structures, which may compromise security on ** [ 0.251018] ** your system. ** [ 0.252018] ** ** [ 0.253017] ** If you see this message and you are not debugging the ** [ 0.254016] ** kernel, report this immediately to your vendor! ** [ 0.255016] ** ** [ 0.256017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257015] ************************************************************* [ 0.258647] NET: Registered protocol family 16 [ 0.259431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.262490] cpuidle: using governor menu [ 0.264907] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.267944] PCI: Using configuration type 1 for base access [ 0.271171] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.283044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.285151] cryptd: max_cpu_qlen set to 1000 [ 0.289021] ACPI: Added _OSI(Module Device) [ 0.290015] ACPI: Added _OSI(Processor Device) [ 0.291016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.292019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.296176] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.299599] ACPI: Interpreter enabled [ 0.300063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.301015] ACPI: Using IOAPIC for interrupt routing [ 0.302110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.303397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.312823] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.313040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.314023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.315084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.317584] acpiphp: Slot [2] registered [ 0.318203] acpiphp: Slot [5] registered [ 0.319174] acpiphp: Slot [6] registered [ 0.320171] acpiphp: Slot [7] registered [ 0.321155] acpiphp: Slot [8] registered [ 0.322182] acpiphp: Slot [9] registered [ 0.323218] acpiphp: Slot [10] registered [ 0.324158] acpiphp: Slot [3] registered [ 0.325248] acpiphp: Slot [4] registered [ 0.326132] acpiphp: Slot [11] registered [ 0.327129] acpiphp: Slot [12] registered [ 0.328196] acpiphp: Slot [13] registered [ 0.329135] acpiphp: Slot [14] registered [ 0.330165] acpiphp: Slot [15] registered [ 0.331135] acpiphp: Slot [16] registered [ 0.332124] acpiphp: Slot [17] registered [ 0.333150] acpiphp: Slot [18] registered [ 0.334209] acpiphp: Slot [19] registered [ 0.335153] acpiphp: Slot [20] registered [ 0.336170] acpiphp: Slot [21] registered [ 0.337131] acpiphp: Slot [22] registered [ 0.338147] acpiphp: Slot [23] registered [ 0.339135] acpiphp: Slot [24] registered [ 0.340132] acpiphp: Slot [25] registered [ 0.341143] acpiphp: Slot [26] registered [ 0.342134] acpiphp: Slot [27] registered [ 0.343177] acpiphp: Slot [28] registered [ 0.344138] acpiphp: Slot [29] registered [ 0.345176] acpiphp: Slot [30] registered [ 0.346136] acpiphp: Slot [31] registered [ 0.347100] PCI host bridge to bus 0000:00 [ 0.348027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.349031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.352033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.355916] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357074] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374909] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.377078] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380904] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416732] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.495573] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.506025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.515025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.535035] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.546119] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.556030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.564025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.582020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.592897] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.603025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.613023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.635023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.647691] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.656024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.666026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.685024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.697016] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.709023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.722020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.739020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.752322] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.754413] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.757404] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.760419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.762288] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.767053] iommu: Default domain type: Passthrough [ 0.768589] SCSI subsystem initialized [ 0.770200] ACPI: bus type USB registered [ 0.771306] usbcore: registered new interface driver usbfs [ 0.773121] usbcore: registered new interface driver hub [ 0.775105] usbcore: registered new device driver usb [ 0.777167] pps_core: LinuxPPS API ver. 1 registered [ 0.778015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.781094] PTP clock support registered [ 0.784008] EDAC MC: Ver: 3.0.0 [ 0.786082] PCI: Using ACPI for IRQ routing [ 0.787991] NetLabel: Initializing [ 0.788014] NetLabel: domain hash size = 128 [ 0.789012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.790142] NetLabel: unlabeled traffic allowed by default [ 0.792097] vgaarb: loaded [ 0.793316] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.795021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.799639] clocksource: Switched to clocksource kvm-clock [ 0.916622] VFS: Disk quotas dquot_6.6.0 [ 0.918439] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.921139] *** VALIDATE ramfs *** [ 0.922396] *** VALIDATE hugetlbfs *** [ 0.924110] pnp: PnP ACPI init [ 0.927457] pnp: PnP ACPI: found 6 devices [ 0.944096] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.946877] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.948635] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.950413] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.952558] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.954476] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.956688] NET: Registered protocol family 2 [ 0.958727] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.963090] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.966080] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.970620] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.973480] TCP: Hash tables configured (established 65536 bind 65536) [ 0.975933] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.978425] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.980889] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983336] NET: Registered protocol family 1 [ 0.985721] RPC: Registered named UNIX socket transport module. [ 0.988174] RPC: Registered udp transport module. [ 0.991432] RPC: Registered tcp transport module. [ 0.993262] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.995876] NET: Registered protocol family 44 [ 0.997727] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.000064] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.002502] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.005053] PCI: CLS 0 bytes, default 64 [ 1.006924] Unpacking initramfs... [ 2.438296] debug: unmapping init [mem 0xffff95ae7cc54000-0xffff95ae7ffbffff] [ 2.442047] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.443912] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.446338] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.966554] Initialise system trusted keyrings [ 2.968504] Key type blacklist registered [ 2.970503] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.979602] zbud: loaded [ 2.982326] *** VALIDATE nfs *** [ 2.983541] *** VALIDATE nfs4 *** [ 2.985049] pstore: using deflate compression [ 2.991685] Platform Keyring initialized [ 3.104850] NET: Registered protocol family 38 [ 3.106656] Key type asymmetric registered [ 3.107570] Asymmetric key parser 'x509' registered [ 3.109019] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.112142] io scheduler mq-deadline registered [ 3.113271] io scheduler kyber registered [ 3.114701] io scheduler bfq registered [ 3.116935] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.119367] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.121508] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.123852] ACPI: Power Button [PWRF] [ 3.129361] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.136140] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.148677] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.155663] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.169304] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.198776] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.228687] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.234556] Non-volatile memory driver v1.3 [ 3.236373] Linux agpgart interface v0.103 [ 3.276265] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.280878] vda: detected capacity change from 0 to 74706944 [ 3.303144] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.306715] vdb: detected capacity change from 0 to 1073741824 [ 3.328448] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.331520] vdc: detected capacity change from 0 to 2621440000 [ 3.349329] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.353223] vdd: detected capacity change from 0 to 2621440000 [ 3.373463] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.376789] vde: detected capacity change from 0 to 4294967296 [ 3.392445] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.395672] vdf: detected capacity change from 0 to 4294967296 [ 3.405153] libphy: Fixed MDIO Bus: probed [ 3.431409] usbcore: registered new interface driver usbserial_generic [ 3.434200] usbserial: USB Serial support registered for generic [ 3.436970] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.442160] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.444515] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.447618] mousedev: PS/2 mouse device common for all mice [ 3.452353] rtc_cmos 00:05: RTC can wake from S4 [ 3.456417] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.460373] rtc_cmos 00:05: registered as rtc0 [ 3.461618] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.464543] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.464672] intel_pstate: CPU model not supported [ 3.469236] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.469532] hid: raw HID events driver (C) Jiri Kosina [ 3.473270] usbcore: registered new interface driver usbhid [ 3.474960] usbhid: USB HID core driver [ 3.476234] drop_monitor: Initializing network drop monitor service [ 3.477989] Initializing XFRM netlink socket [ 3.479407] NET: Registered protocol family 10 [ 3.482529] Segment Routing with IPv6 [ 3.484194] NET: Registered protocol family 17 [ 3.486385] mpls_gso: MPLS GSO support [ 3.492663] RAS: Correctable Errors collector initialized. [ 3.495288] AVX version of gcm_enc/dec engaged. [ 3.496504] AES CTR mode by8 optimization enabled [ 3.596378] sched_clock: Marking stable (3596329808, 0)->(4605411781, -1009081973) [ 3.600805] registered taskstats version 1 [ 3.602793] Loading compiled-in X.509 certificates [ 3.605427] zswap: loaded using pool lzo/zbud [ 3.627036] Key type big_key registered [ 3.643089] Key type encrypted registered [ 3.644618] ima: No TPM chip found, activating TPM-bypass! [ 3.646649] ima: Allocated hash algorithm: sha1 [ 3.648256] ima: No architecture policies found [ 3.650090] evm: Initialising EVM extended attributes: [ 3.652059] evm: security.selinux [ 3.653272] evm: security.ima [ 3.654478] evm: security.capability [ 3.655787] evm: HMAC attrs: 0x1 [ 3.658510] rtc_cmos 00:05: setting system clock to 2026-08-13 17:16:48 UTC (1786641408) [ 3.664840] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.668026] debug: unmapping init [mem 0xffffffffac782000-0xffffffffaca58fff] [ 3.678105] Write protecting the kernel read-only data: 28672k [ 3.681966] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.684633] debug: unmapping init [mem 0xffffffffab714000-0xffffffffab7fffff] [ 3.729053] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.738062] systemd[1]: Detected virtualization kvm. [ 3.740130] systemd[1]: Detected architecture x86-64. [ 3.742161] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.770684] systemd[1]: No hostname configured. [ 3.772633] systemd[1]: Set hostname to . [ 3.774721] random: systemd: uninitialized urandom read (16 bytes read) [ 3.777448] systemd[1]: Initializing machine ID from random generator. [ 3.925911] random: systemd: uninitialized urandom read (16 bytes read) [ 3.928694] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.936174] random: systemd: uninitialized urandom read (16 bytes read) [ 3.939528] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.944296] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.648258] device-mapper: uevent: version 1.0.3 [ 4.650534] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.373067] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.473223] virtio_net virtio0 ens2: renamed from eth0 [ 5.539761] scsi host0: ata_piix [ 5.569363] scsi host1: ata_piix [ 5.571496] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.573886] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.053957] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.164281] random: crng init done [ 10.165861] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.716688] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.904087] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.166268] SELinux: Disabled at runtime. [ 12.228338] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.236484] systemd[1]: Detected virtualization kvm. [ 12.238079] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.708043] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.711917] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.717634] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.721655] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.724993] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.731465] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.736310] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 12.799560] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.194043] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.431465] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.493326] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.566787] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.588740] EDAC sbridge: Ver: 1.1.2 [ 15.309746] Key type dns_resolver registered [ 15.671697] NFS: Registering the id_resolver key type [ 15.674198] Key type id_resolver registered [ 15.676145] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg209-server login: [ 28.121046] hrtimer: interrupt took 2763230 ns [ 69.948619] libcfs: loading out-of-tree module taints kernel. [ 70.036874] Key type ._llcrypt registered [ 70.043674] Key type .llcrypt registered [ 70.199788] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 86.096660] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 87.425697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.450431] alg: No test for adler32 (adler32-zlib) [ 88.877666] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 89.808595] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 91.640945] Key type lgssc registered [ 93.203197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 109.229562] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.595672] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 156.447673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.489253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.681459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.707352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.766928] Lustre: lustre-MDT0000: new disk, initializing [ 157.826357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.851578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.044511] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.958162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.105267] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.175964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.183635] Lustre: Skipped 1 previous similar message [ 180.431113] Lustre: lustre-MDT0001: new disk, initializing [ 180.543915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.562345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.567522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.439567] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.617252] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 203.543968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.899825] Lustre: lustre-OST0000: new disk, initializing [ 203.905466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.917700] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.986841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.788366] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.125356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 214.144743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 214.264275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 228.747649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.080949] Lustre: lustre-OST0001: new disk, initializing [ 229.086638] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.104995] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.257548] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.096698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.120882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.192379] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.995528] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.950538] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.941671] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 263.918865] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.345636] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing yml_node [ 272.557840] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 274.452556] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 276.297932] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 277.703083] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 13:21:21 EDT 2026 [ 291.798436] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 293.135895] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 308.709640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 308.724195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.738793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.289222] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.291754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.307564] Lustre: Skipped 2 previous similar messages [ 312.335444] Lustre: Skipped 3 previous similar messages [ 314.883387] Lustre: server umount lustre-MDT0000 complete [ 318.574883] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641723 with bad export cookie 11053995819536475497 [ 318.578136] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.586392] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 322.532071] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.534094] LustreError: 8755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.542750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.570381] LustreError: 8755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 323.554161] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.566883] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 324.959472] Lustre: server umount lustre-MDT0001 complete [ 328.978975] Lustre: server umount lustre-OST0000 complete [ 332.959661] Lustre: server umount lustre-OST0001 complete [ 338.841215] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 346.993828] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 383.353848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.863788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 383.902593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.023898] Lustre: lustre-MDT0000: new disk, initializing [ 384.101062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.121372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.528836] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 398.346023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.468511] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 398.527703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 398.532676] Lustre: Skipped 1 previous similar message [ 398.669320] Lustre: lustre-MDT0001: new disk, initializing [ 398.773712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 398.810039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 398.822363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 402.949489] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 411.722192] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 417.933399] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.618559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 424.948796] Lustre: lustre-OST0000: new disk, initializing [ 424.952350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 424.963858] Lustre: 17039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.061023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.792146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.804289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.948234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.442595] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 441.665213] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 446.525144] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 446.805439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.903993] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 452.135330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.707057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.718054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.724912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.751435] Lustre: Skipped 2 previous similar messages [ 457.791452] Lustre: Skipped 4 previous similar messages [ 460.856120] Lustre: server umount lustre-OST0000 complete [ 465.381153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 465.386307] LustreError: Skipped 1 previous similar message [ 465.395874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.413925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 470.679373] Lustre: server umount lustre-MDT0000 complete [ 473.065451] LustreError: 15075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.093387] LustreError: 15075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 474.543960] LustreError: 15866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641879 with bad export cookie 11053995819536477030 [ 474.545562] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.567498] LustreError: 15866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 475.028502] Lustre: server umount lustre-MDT0001 complete [ 478.797137] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 13:24:42 (1786641882) [ 485.685754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.089387] LustreError: 19304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.152451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.289590] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 491.504551] LustreError: 19305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.559755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.316563] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 511.439726] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 519.184299] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 527.751483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.071651] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 545.720651] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 554.983217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.987269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 555.017065] Lustre: Skipped 1 previous similar message [ 555.020602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 555.020624] Lustre: Skipped 2 previous similar messages [ 558.833938] Lustre: server umount lustre-OST0000 complete [ 571.261392] Lustre: server umount lustre-MDT0000 complete [ 574.438670] LustreError: 19304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.458832] LustreError: 19304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 575.398611] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641980 with bad export cookie 11053995819536477912 [ 575.420294] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.006256] Lustre: server umount lustre-MDT0001 complete [ 591.822968] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 596.988258] Key type lgssc unregistered [ 597.430744] LNet: 22840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.440552] LNetError: 22840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.484322] LNet: Removed LNI 192.168.202.109@tcp [ 599.093274] Key type .llcrypt unregistered [ 599.095101] Key type ._llcrypt unregistered [ 627.883170] Key type ._llcrypt registered [ 627.884887] Key type .llcrypt registered [ 628.075813] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 629.480222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 629.535814] alg: No test for adler32 (adler32-zlib) [ 630.739523] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 631.066791] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 632.744180] Key type lgssc registered [ 634.358483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 642.427794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 642.459694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.026608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.384201] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 653.288721] LustreError: 24256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.316858] LustreError: 24256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 658.235376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.402495] LustreError: 24257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.779961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 663.689726] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 672.063705] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 678.092307] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 685.902861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 686.280107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.390945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 691.525944] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 698.947447] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 715.252087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.256621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 715.277668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 715.296626] Lustre: Skipped 1 previous similar message [ 720.357166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.534670] Lustre: server umount lustre-OST0000 complete [ 725.476051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.487343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 725.512966] Lustre: Skipped 1 previous similar message [ 725.514265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.514270] Lustre: Skipped 1 previous similar message [ 725.564573] LustreError: Skipped 1 previous similar message [ 730.603070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 730.619026] Lustre: Skipped 2 previous similar messages [ 735.714955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.745457] Lustre: Skipped 1 previous similar message [ 739.296124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 739.728899] Lustre: server umount lustre-MDT0000 complete [ 740.834674] LustreError: 24252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.844819] LustreError: 24252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 744.156329] LustreError: 24991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642148 with bad export cookie 16154898888923773792 [ 744.167417] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.185908] LustreError: 24991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 745.014965] Lustre: server umount lustre-MDT0001 complete [ 759.352840] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 762.473145] Key type lgssc unregistered [ 762.766574] LNet: 28238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.777089] LNetError: 28238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.814051] LNet: Removed LNI 192.168.202.109@tcp [ 764.929251] Key type .llcrypt unregistered [ 764.935598] Key type ._llcrypt unregistered [ 790.574530] Key type ._llcrypt registered [ 790.577615] Key type .llcrypt registered [ 790.677893] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 791.624412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 791.658417] alg: No test for adler32 (adler32-zlib) [ 792.867876] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 793.258126] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 794.928517] Key type lgssc registered [ 796.601518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 804.345715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 804.363662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.630766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 809.492721] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 816.099691] LustreError: 29652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 816.118034] LustreError: 29652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 816.216318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.492139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 820.141398] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 825.724458] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 830.892476] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 845.260935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 845.604803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.764877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 853.366658] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 859.917126] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 871.393882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.402081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.409330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.420453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.430269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.463520] Lustre: Skipped 1 previous similar message [ 876.519403] Lustre: server umount lustre-OST0000 complete [ 882.659769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.661788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 882.672667] Lustre: Skipped 1 previous similar message [ 882.686487] Lustre: Skipped 1 previous similar message [ 886.405995] Lustre: server umount lustre-MDT0000 complete [ 887.778221] LustreError: 29651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.803854] LustreError: 29651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 889.787209] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642294 with bad export cookie 3631372225148924727 [ 889.792187] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 889.799094] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 890.228779] Lustre: server umount lustre-MDT0001 complete [ 900.400506] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 902.694385] Key type lgssc unregistered [ 902.958642] LNet: 33532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.962687] LNetError: 33532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.974917] LNet: Removed LNI 192.168.202.109@tcp [ 903.526191] Key type .llcrypt unregistered [ 903.528324] Key type ._llcrypt unregistered [ 926.416891] Key type ._llcrypt registered [ 926.419671] Key type .llcrypt registered [ 926.536911] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 927.728682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 927.784408] alg: No test for adler32 (adler32-zlib) [ 928.805672] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 929.098578] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 930.784164] Key type lgssc registered [ 931.844733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 938.506247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 938.517946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.887132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.539193] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 949.222483] LustreError: 34942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.241995] LustreError: 34942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 951.832516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.157384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 956.464471] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 962.999848] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 968.112922] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 973.228475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 973.565029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.409587] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 984.070748] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 984.835146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 997.345970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 997.350881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.372282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 997.860228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 997.873803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.889673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1000.427290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1000.448797] Lustre: Skipped 1 previous similar message [ 1002.091673] Lustre: server umount lustre-OST0000 complete [ 1008.614344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1008.620503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.639380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.664825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.877198] Lustre: server umount lustre-MDT0000 complete [ 1015.204313] LustreError: 34923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642420 with bad export cookie 12969911042312963690 [ 1015.209355] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1015.213809] LustreError: 34923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1015.519904] Lustre: server umount lustre-MDT0001 complete [ 1025.994557] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1028.715674] Key type lgssc unregistered [ 1028.933864] LNet: 38481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.939251] LNetError: 38481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.960214] LNet: Removed LNI 192.168.202.109@tcp [ 1029.888250] Key type .llcrypt unregistered [ 1029.892617] Key type ._llcrypt unregistered [ 1052.870752] Key type ._llcrypt registered [ 1052.872704] Key type .llcrypt registered [ 1052.977268] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1053.945275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.076586] alg: No test for adler32 (adler32-zlib) [ 1055.079613] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1055.257498] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1056.904354] Key type lgssc registered [ 1057.821734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.604410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1064.623722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.092833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.564048] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1075.683842] LustreError: 39894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.706294] LustreError: 39894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1080.804475] LustreError: 39895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.025931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.467504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1085.857833] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1095.157887] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1100.723502] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1108.334650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1108.902227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.960564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1114.474608] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1122.269805] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1131.491915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1131.495675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.532325] LustreError: 41719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.545090] LustreError: 41719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1131.760080] Lustre: server umount lustre-OST0000 complete [ 1151.970472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.993327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.007409] Lustre: Skipped 2 previous similar messages [ 1152.018041] Lustre: Skipped 1 previous similar message [ 1153.063434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.076568] Lustre: Skipped 1 previous similar message [ 1158.138295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.140826] Lustre: Skipped 1 previous similar message [ 1163.242291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.258879] Lustre: Skipped 1 previous similar message [ 1166.304121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1166.480836] Lustre: server umount lustre-MDT0000 complete [ 1168.354708] LustreError: 40932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1168.400328] LustreError: 40932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1170.317829] LustreError: 39875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642575 with bad export cookie 9225955408834711280 [ 1170.329703] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1176.594339] Lustre: server umount lustre-MDT0001 complete [ 1186.051583] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1189.308658] Key type lgssc unregistered [ 1189.680754] LNet: 43606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.685952] LNetError: 43606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.714523] LNet: Removed LNI 192.168.202.109@tcp [ 1190.681101] Key type .llcrypt unregistered [ 1190.683479] Key type ._llcrypt unregistered [ 1222.730913] Key type ._llcrypt registered [ 1222.733197] Key type .llcrypt registered [ 1222.913045] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1224.285773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1224.365147] alg: No test for adler32 (adler32-zlib) [ 1225.508351] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1225.815832] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1227.546502] Key type lgssc registered [ 1229.399265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1235.986534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1236.014105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.410277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.956962] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1247.717462] LustreError: 45021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.740162] LustreError: 45021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1250.935426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.262906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1256.401327] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1264.393409] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1270.180577] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1275.956819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1276.282846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.445451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1283.519917] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1290.256787] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1297.891878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.900691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1297.904726] Lustre: Skipped 1 previous similar message [ 1297.916377] Lustre: Skipped 1 previous similar message [ 1302.500858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1302.515779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.527995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.531909] Lustre: Skipped 1 previous similar message [ 1308.136606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.146860] Lustre: Skipped 6 previous similar messages [ 1312.224503] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1312.495984] Lustre: server umount lustre-MDT0000 complete [ 1313.252808] LustreError: 45020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.298447] LustreError: 45020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1316.042880] LustreError: 45002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642720 with bad export cookie 351567968824528737 [ 1316.051727] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.063110] LustreError: 45002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1318.370286] LustreError: 45017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.374467] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.384521] LustreError: 45017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1318.414235] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1318.418824] Lustre: Skipped 1 previous similar message [ 1321.378878] LustreError: 45016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.388816] LustreError: 45016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1322.419995] Lustre: server umount lustre-MDT0001 complete [ 1337.387819] Lustre: server umount lustre-OST0000 complete [ 1353.127831] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1355.997958] Key type lgssc unregistered [ 1356.365786] LNet: 48883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.373168] LNetError: 48883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.391842] LNet: Removed LNI 192.168.202.109@tcp [ 1357.142785] Key type .llcrypt unregistered [ 1357.144624] Key type ._llcrypt unregistered [ 1380.994660] Key type ._llcrypt registered [ 1380.997532] Key type .llcrypt registered [ 1381.108224] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1382.334765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1382.453287] alg: No test for adler32 (adler32-zlib) [ 1383.774703] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1384.081845] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1385.760199] Key type lgssc registered [ 1387.107331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1394.148860] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1394.179996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.760974] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1414.880357] LustreError: 50273:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 1441.568260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.085853] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1456.552813] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1623.310863] Lustre: server umount lustre-OST0000 complete [ 1639.191326] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1641.449270] Key type lgssc unregistered [ 1641.737613] LNet: 51949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.741492] LNetError: 51949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.753250] LNet: Removed LNI 192.168.202.109@tcp [ 1642.226750] Key type .llcrypt unregistered [ 1642.228455] Key type ._llcrypt unregistered [ 1663.140399] Key type ._llcrypt registered [ 1663.144296] Key type .llcrypt registered [ 1663.226536] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1664.254842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1664.373106] alg: No test for adler32 (adler32-zlib) [ 1665.393509] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1665.673615] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1667.321126] Key type lgssc registered [ 1668.286265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1675.497135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1675.513394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.910421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.008346] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1686.506702] LustreError: 53363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.520134] LustreError: 53363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1689.921889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.309663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1696.421352] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1704.833410] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1710.888266] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1717.207780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1717.533976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.660393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1723.114626] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1730.068840] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1735.343344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1735.352019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.459768] Lustre: server umount lustre-OST0000 complete [ 1741.793387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1741.804704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.818287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1741.821987] Lustre: Skipped 1 previous similar message [ 1745.891938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.896481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.069231] Lustre: server umount lustre-MDT0000 complete [ 1751.009595] LustreError: 55437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1751.027653] LustreError: 55437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1751.211815] LustreError: 53343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643156 with bad export cookie 13449397350295675682 [ 1751.217467] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1751.222727] LustreError: 53343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1751.870524] Lustre: server umount lustre-MDT0001 complete [ 1762.662872] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1765.340862] Key type lgssc unregistered [ 1765.607673] LNet: 56899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.618853] LNetError: 56899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.634182] LNet: Removed LNI 192.168.202.109@tcp [ 1766.366278] Key type .llcrypt unregistered [ 1766.369228] Key type ._llcrypt unregistered [ 1785.182554] Key type ._llcrypt registered [ 1785.184561] Key type .llcrypt registered [ 1785.327184] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1786.383871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1786.467810] alg: No test for adler32 (adler32-zlib) [ 1787.485315] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1787.660528] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1789.297084] Key type lgssc registered [ 1790.250836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.519890] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.538935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.064411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1817.186483] LustreError: 58264:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 1844.067299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.432682] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1856.539940] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1862.760513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.311329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1863.403683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1866.799241] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1868.780636] LustreError: 58288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.820994] LustreError: 58288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1873.895809] LustreError: 58289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.022791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.653859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1882.867709] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1891.962833] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1898.874302] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1903.076419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.084298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.102936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.625988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.626463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1904.636655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.888407] Lustre: server umount lustre-OST0000 complete [ 1930.020453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.035097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.051077] Lustre: Skipped 1 previous similar message [ 1930.057759] Lustre: Skipped 1 previous similar message [ 1935.350226] LustreError: 60083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.390196] LustreError: 60083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1935.898300] Lustre: server umount lustre-MDT0000 complete [ 1940.449576] LustreError: 59300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.473205] LustreError: 59300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1940.905154] LustreError: 58271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643345 with bad export cookie 14548220545835261280 [ 1940.920517] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1940.928683] LustreError: 58271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1955.296337] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1955.830902] Lustre: server umount lustre-MDT0001 complete [ 1966.816803] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1969.499788] Key type lgssc unregistered [ 1969.722237] LNet: 62015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.729698] LNetError: 62015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.793649] LNet: Removed LNI 192.168.202.109@tcp [ 1971.606908] Key type .llcrypt unregistered [ 1971.610667] Key type ._llcrypt unregistered [ 1993.473928] Key type ._llcrypt registered [ 1993.476238] Key type .llcrypt registered [ 1993.558889] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1994.978037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1995.003183] alg: No test for adler32 (adler32-zlib) [ 1996.056716] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1996.309307] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1997.984312] Key type lgssc registered [ 1999.054104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2005.570831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2005.586130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2006.980053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.964849] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2016.227406] LustreError: 63409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.242040] LustreError: 63409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2018.349410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.685646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2022.794411] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2029.590861] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2036.199637] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2045.079261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2045.511260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.658620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2051.936870] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2059.516150] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2077.153924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2077.165974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.185683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2078.521466] Lustre: server umount lustre-OST0000 complete [ 2084.839433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.840900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.848410] Lustre: Skipped 1 previous similar message [ 2084.860941] Lustre: Skipped 1 previous similar message [ 2087.869575] Lustre: server umount lustre-MDT0000 complete [ 2089.967041] LustreError: 64458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.978643] LustreError: 64458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2092.114824] LustreError: 63389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643496 with bad export cookie 7204111022469485900 [ 2092.136393] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.018586] Lustre: server umount lustre-MDT0001 complete [ 2115.562254] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2119.793588] Key type lgssc unregistered [ 2120.419645] LNet: 66971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.439743] LNetError: 66971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.492806] LNet: Removed LNI 192.168.202.109@tcp [ 2122.151316] Key type .llcrypt unregistered [ 2122.159300] Key type ._llcrypt unregistered [ 2149.335381] Key type ._llcrypt registered [ 2149.338234] Key type .llcrypt registered [ 2149.423409] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2150.727718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.819831] alg: No test for adler32 (adler32-zlib) [ 2152.024509] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2152.213075] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2153.832411] Key type lgssc registered [ 2155.243559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2164.576312] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 13:52:48 (1786643568) [ 2170.496660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2170.511637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.990107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.627927] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2182.130652] LustreError: 68676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.162281] LustreError: 68676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2187.246349] LustreError: 68677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.213120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.869548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2194.029165] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2202.507427] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2208.795561] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2215.441908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2215.748761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.817178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2221.352524] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2227.798637] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2235.819503] Lustre: Failing over lustre-MDT0000 [ 2236.039973] Lustre: server umount lustre-MDT0000 complete [ 2238.434681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.445825] LustreError: 68672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.447242] Lustre: Skipped 1 previous similar message [ 2238.471319] LustreError: 68672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2243.553726] LustreError: 68677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.571576] LustreError: 68677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2243.789634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.880299] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2243.973321] Lustre: *** cfs_fail_loc=135, val=0*** [ 2244.043541] LustreError: 71763:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2244.050674] LustreError: 71763:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -2 [ 2244.059428] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2244.059428] [ 2244.066436] LustreError: MGC192.168.202.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2244.076441] LustreError: 71752:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2244.081455] LustreError: 71752:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2244.090865] LustreError: 71752:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2244.095335] LustreError: 71752:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2244.115421] Lustre: server umount lustre-MDT0000 complete [ 2244.118462] LustreError: 71752:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2248.675590] LustreError: 70725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.704574] LustreError: 70725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2249.920344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.050717] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.254262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.298416] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2254.765955] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2254.947608] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2255.342261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2255.369695] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2255.408905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2264.038925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.050672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.067464] Lustre: Skipped 1 previous similar message [ 2264.078177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.574597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.582919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.647968] Lustre: server umount lustre-OST0000 complete [ 2275.809839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.825159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2275.836383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.836391] Lustre: Skipped 1 previous similar message [ 2277.101501] Lustre: server umount lustre-MDT0000 complete [ 2280.137405] LustreError: 68657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643684 with bad export cookie 9992580696636379869 [ 2280.140156] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.150929] LustreError: 68657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2280.423374] Lustre: server umount lustre-MDT0001 complete [ 2289.658736] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2292.250514] Key type lgssc unregistered [ 2292.510074] LNet: 73544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.521312] LNetError: 73544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.544079] LNet: Removed LNI 192.168.202.109@tcp [ 2293.419146] Key type .llcrypt unregistered [ 2293.423584] Key type ._llcrypt unregistered [ 2315.682567] Key type ._llcrypt registered [ 2315.685707] Key type .llcrypt registered [ 2315.899717] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2317.184156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.205208] alg: No test for adler32 (adler32-zlib) [ 2318.271735] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2318.518721] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2320.152433] Key type lgssc registered [ 2321.045074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2327.762708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2327.786806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.292588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.939422] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2339.813037] LustreError: 74953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.828926] LustreError: 74953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2342.160549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.433582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.475450] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2353.681320] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2361.515650] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2369.966678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2370.312813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.413834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2376.500259] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2383.221811] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2393.147740] Lustre: Failing over lustre-MDT0000 [ 2393.345200] Lustre: server umount lustre-MDT0000 complete [ 2393.569546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2393.575153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.586466] LustreError: 74954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.301355] LustreError: 74950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.180544] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.204489] Lustre: Skipped 1 previous similar message [ 2401.379490] LustreError: 77446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.396397] LustreError: 77446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2402.473636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.706990] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2402.961947] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2403.143640] LustreError: 78042:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2403.157777] LustreError: 78042:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -9 [ 2403.166977] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2403.166977] [ 2403.175407] LustreError: MGC192.168.202.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2403.198660] LustreError: 78031:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2403.207578] LustreError: 78031:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2403.239603] LustreError: 78031:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2403.249984] LustreError: 78031:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2403.274816] Lustre: server umount lustre-MDT0000 complete [ 2403.280050] LustreError: 78031:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2406.498686] LustreError: 74949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.507233] LustreError: 74949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2409.913117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.031523] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2410.038559] LustreError: 74945:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2410.273577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.310824] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2412.610970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2415.022444] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2415.642904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2415.690503] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2415.762204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2423.532398] Lustre: server umount lustre-OST0000 complete [ 2430.946379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.949076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.964843] Lustre: Skipped 1 previous similar message [ 2430.978422] Lustre: Skipped 1 previous similar message [ 2436.066748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.096158] Lustre: 74266:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643828/real 1786643828] req@00000000338740d4 x1873431728533120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786643844 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2440.127377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.141267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.147401] Lustre: Skipped 2 previous similar messages [ 2441.184141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2441.186061] Lustre: 74265:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643830/real 1786643830] req@00000000c978d2b3 x1873431728533632/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786643846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2441.436650] Lustre: server umount lustre-MDT0000 complete [ 2444.982498] LustreError: 75688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643849 with bad export cookie 1036443407317220180 [ 2444.985518] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2444.992801] LustreError: 75688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2445.343797] Lustre: server umount lustre-MDT0001 complete [ 2454.768226] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2457.252494] Key type lgssc unregistered [ 2457.477874] LNet: 79825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.484236] LNetError: 79825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.497407] LNet: Removed LNI 192.168.202.109@tcp [ 2458.301081] Key type .llcrypt unregistered [ 2458.303562] Key type ._llcrypt unregistered [ 2468.422969] Key type ._llcrypt registered [ 2468.425539] Key type .llcrypt registered [ 2468.501147] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2468.744369] Key type .llcrypt unregistered [ 2468.747566] Key type ._llcrypt unregistered [ 2488.189638] Key type ._llcrypt registered [ 2488.192047] Key type .llcrypt registered [ 2488.338535] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2489.762364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2489.902387] alg: No test for adler32 (adler32-zlib) [ 2491.077562] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2491.397537] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2493.096508] Key type lgssc registered [ 2494.194960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2501.013877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2501.027812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.285652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.854113] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2512.870527] LustreError: 81712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.886973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.899213] LustreError: 81712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2513.284976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2517.504899] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2524.268564] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2529.070275] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2534.223316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2534.440545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2539.605714] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2544.813356] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2547.779225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2563.043115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2563.050273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.066668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.083106] Lustre: Skipped 1 previous similar message [ 2563.098631] Lustre: Skipped 1 previous similar message [ 2565.264502] Lustre: server umount lustre-OST0000 complete [ 2569.697905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2569.701834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.711931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.282911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.287167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.614338] Lustre: server umount lustre-MDT0000 complete [ 2578.324453] LustreError: 81693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643983 with bad export cookie 3833562531880027027 [ 2578.334981] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2578.335788] LustreError: 81693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2578.865038] Lustre: server umount lustre-MDT0001 complete [ 2588.388632] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2590.648719] Key type lgssc unregistered [ 2590.892673] LNet: 85264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.898174] LNetError: 85264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2590.916252] LNet: Removed LNI 192.168.202.109@tcp [ 2591.657868] Key type .llcrypt unregistered [ 2591.659592] Key type ._llcrypt unregistered [ 2613.121404] Key type ._llcrypt registered [ 2613.122687] Key type .llcrypt registered [ 2613.199258] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2614.519262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2614.556820] alg: No test for adler32 (adler32-zlib) [ 2615.605780] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2615.807096] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2617.424710] Key type lgssc registered [ 2618.339207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2625.158982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2625.176038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.626700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.510535] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2636.775175] LustreError: 86659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.798544] LustreError: 86659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2637.207342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.474757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2641.471439] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2647.938532] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2652.448376] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2659.068420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.463856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.839929] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2671.501681] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2671.731522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2678.241185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2678.257183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.276371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2679.720147] Lustre: server umount lustre-OST0000 complete [ 2683.874249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2683.887133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.909049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.459225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.461988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.577578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2692.586448] Lustre: Skipped 2 previous similar messages [ 2693.765078] Lustre: server umount lustre-MDT0000 complete [ 2697.578566] LustreError: 86638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644102 with bad export cookie 2184943084060990872 [ 2697.581776] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2697.587952] LustreError: 86638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2697.894197] Lustre: server umount lustre-MDT0001 complete [ 2707.272242] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2709.542285] Key type lgssc unregistered [ 2709.801898] LNet: 90205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2709.811964] LNetError: 90205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2709.832889] LNet: Removed LNI 192.168.202.109@tcp [ 2710.680392] Key type .llcrypt unregistered [ 2710.682724] Key type ._llcrypt unregistered [ 2729.994196] Key type ._llcrypt registered [ 2730.000246] Key type .llcrypt registered [ 2730.097900] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2730.890881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.061301] alg: No test for adler32 (adler32-zlib) [ 2732.072328] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2732.306751] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2733.969653] Key type lgssc registered [ 2734.907391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.516132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.531615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.943810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.697751] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2752.486768] LustreError: 91598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.498686] LustreError: 91598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2754.310239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.590714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2758.693933] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2765.734893] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2770.247674] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2775.893052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.170042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2781.705252] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2787.489462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2787.864808] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2798.052506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.064616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2798.065960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.083944] Lustre: Skipped 1 previous similar message [ 2799.744789] Lustre: server umount lustre-OST0000 complete [ 2803.172101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.188292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.194094] Lustre: Skipped 2 previous similar messages [ 2803.219696] Lustre: Skipped 1 previous similar message [ 2808.290233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2809.227751] Lustre: server umount lustre-MDT0000 complete [ 2812.732987] LustreError: 91623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644217 with bad export cookie 13714156388044419353 [ 2812.741241] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2812.754545] LustreError: 91623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2813.125778] Lustre: server umount lustre-MDT0001 complete [ 2823.095703] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2825.604506] Key type lgssc unregistered [ 2825.945598] LNet: 95134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2825.956674] LNetError: 95134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2826.012384] LNet: Removed LNI 192.168.202.109@tcp [ 2827.198272] Key type .llcrypt unregistered [ 2827.200096] Key type ._llcrypt unregistered [ 2852.453896] Key type ._llcrypt registered [ 2852.457546] Key type .llcrypt registered [ 2852.622049] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2854.343389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2854.396074] alg: No test for adler32 (adler32-zlib) [ 2855.374944] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2855.566966] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2857.248224] Key type lgssc registered [ 2858.555719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2865.141337] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2865.152138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.608479] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2885.728427] LustreError: 96521:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 2912.416311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.261599] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2924.591623] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2929.043325] Lustre: server umount lustre-OST0000 complete [ 2936.038719] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:05:39 (1786644339) [ 2941.875304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.283483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.568244] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2947.560014] LustreError: 98160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.574664] LustreError: 98160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2952.685047] LustreError: 98159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.837420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.140748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2957.561527] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2964.060743] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2969.211324] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2976.071582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.350676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.513377] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2987.716297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2989.360663] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3003.370256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.372932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3003.399444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.417313] Lustre: Skipped 1 previous similar message [ 3006.055969] Lustre: server umount lustre-OST0000 complete [ 3010.529380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3010.540719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.554861] Lustre: Skipped 1 previous similar message [ 3010.565760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.601258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.616190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.722974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.734509] Lustre: Skipped 2 previous similar messages [ 3023.848535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.859603] Lustre: Skipped 1 previous similar message [ 3024.355473] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3024.508156] Lustre: server umount lustre-MDT0000 complete [ 3028.533925] LustreError: 98140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644433 with bad export cookie 7623787620343222610 [ 3028.536037] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.545859] LustreError: 98140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3029.106923] Lustre: server umount lustre-MDT0001 complete [ 3039.098255] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3041.568692] Key type lgssc unregistered [ 3041.796210] LNet: 101856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3041.802381] LNetError: 101856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3041.820749] LNet: Removed LNI 192.168.202.109@tcp [ 3042.741451] Key type .llcrypt unregistered [ 3042.759946] Key type ._llcrypt unregistered [ 3089.121423] Key type ._llcrypt registered [ 3089.123195] Key type .llcrypt registered [ 3089.242313] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3090.742590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3090.976672] alg: No test for adler32 (adler32-zlib) [ 3092.156337] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3092.432627] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3094.208202] Key type lgssc registered [ 3095.704376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3103.441940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3103.467449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.037953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3108.410250] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3114.481559] LustreError: 104045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.513031] LustreError: 104045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3118.370408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.637153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3124.532854] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3133.070557] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3140.797307] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3149.270945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.689606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.033768] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3163.015777] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3174.288681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3184.610791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.614455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3184.632331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.705796] Lustre: server umount lustre-OST0000 complete [ 3194.856637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.876419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.881728] Lustre: Skipped 2 previous similar messages [ 3194.910612] Lustre: Skipped 2 previous similar messages [ 3199.183182] Lustre: server umount lustre-MDT0000 complete [ 3199.988169] LustreError: 104045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.016337] LustreError: 104045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3203.327826] LustreError: 104026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644608 with bad export cookie 5824039522184496193 [ 3203.331818] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.341129] LustreError: 104026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3203.776598] Lustre: server umount lustre-MDT0001 complete [ 3215.631622] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3219.625823] Key type lgssc unregistered [ 3220.046747] LNet: 107603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3220.068278] LNetError: 107603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3221.103674] LNet: Removed LNI 192.168.202.109@tcp [ 3222.257760] Key type .llcrypt unregistered [ 3222.260472] Key type ._llcrypt unregistered [ 3241.368404] Key type ._llcrypt registered [ 3241.370916] Key type .llcrypt registered [ 3241.479924] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3243.017502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3243.111492] alg: No test for adler32 (adler32-zlib) [ 3244.607542] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3245.059646] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3246.736208] Key type lgssc registered [ 3248.696931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3257.949177] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3257.974444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.568497] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3278.688337] LustreError: 108847:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 3305.446057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.204906] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3319.689725] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3326.050743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.190891] LustreError: 108182:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001f96da0b x1873432699472128/t0(0) o250->MGC192.168.202.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3326.368897] LustreError: MGC192.168.202.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3326.380167] LustreError: 109873:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3326.387803] LustreError: 109873:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3326.393988] LustreError: 109873:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3326.514362] Lustre: server umount lustre-MDT0000 complete [ 3326.519387] LustreError: 109873:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3347.936157] Lustre: 108859:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786644736/real 1786644736] req@000000008c747b4d x1873432699473280/t0(0) o101->MGC192.168.202.109@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1786644752 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3347.975850] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3348.009514] LustreError: 108859:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3368.499978] Lustre: server umount lustre-OST0000 complete [ 3377.603798] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3388.170702] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3435.011291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.228494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3435.245496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3435.309893] Lustre: lustre-MDT0000: new disk, initializing [ 3435.381322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.399224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3440.909981] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3453.432803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.627519] Lustre: 112672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3453.696823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3453.705903] Lustre: Skipped 1 previous similar message [ 3453.812210] Lustre: lustre-MDT0001: new disk, initializing [ 3453.893281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.931889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3453.942098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3458.428409] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3466.856721] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3471.681622] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3477.838569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3478.173875] Lustre: lustre-OST0000: new disk, initializing [ 3478.183610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3478.191200] Lustre: 114638:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3478.274679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.756835] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3488.383058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3488.401615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3488.618368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3493.621624] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3499.090161] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3499.349667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3503.368465] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3503.473753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3509.226814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.227704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.232427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.245287] Lustre: Skipped 1 previous similar message [ 3509.272147] LustreError: Skipped 1 previous similar message [ 3511.079446] Lustre: server umount lustre-OST0000 complete [ 3515.362277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3515.372710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.385714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.396255] Lustre: Skipped 1 previous similar message [ 3519.463840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.468506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.495348] Lustre: Skipped 1 previous similar message [ 3524.580123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.587287] Lustre: Skipped 1 previous similar message [ 3529.186559] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3529.424318] Lustre: server umount lustre-MDT0000 complete [ 3532.860494] LustreError: 112663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644937 with bad export cookie 16302360004817189871 [ 3532.865235] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3532.870803] LustreError: 112663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3533.328878] Lustre: server umount lustre-MDT0001 complete [ 3542.297925] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:15:45 (1786644945) [ 3546.385221] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3567.900027] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3580.425199] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3630.199777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.501745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3630.544572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3630.676280] Lustre: lustre-MDT0000: new disk, initializing [ 3630.868932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.917137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3637.460515] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3648.787251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.909775] Lustre: 119542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3648.975327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3648.979325] Lustre: Skipped 1 previous similar message [ 3649.063769] Lustre: lustre-MDT0001: new disk, initializing [ 3649.181474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3649.232167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3649.242059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3653.361742] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3661.968883] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3667.438602] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3673.950522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3674.316944] Lustre: lustre-OST0000: new disk, initializing [ 3674.323134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3674.338733] Lustre: 121509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3674.497444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.877644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3680.896919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3681.002332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3681.242862] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3690.952362] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3695.523595] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3695.731138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3700.692819] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3700.971513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3706.867075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3706.879920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.882897] LustreError: Skipped 1 previous similar message [ 3706.884557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.926855] Lustre: Skipped 1 previous similar message [ 3717.091745] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3717.095372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.125751] Lustre: Skipped 4 previous similar messages [ 3717.379978] Lustre: server umount lustre-OST0000 complete [ 3722.215206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.231243] Lustre: server umount lustre-MDT0000 complete [ 3732.202087] LustreError: 119535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645137 with bad export cookie 16302360004817190753 [ 3732.210116] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3732.213082] LustreError: 119535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3732.550736] Lustre: server umount lustre-MDT0001 complete [ 3742.462038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.860268] LustreError: 123738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.894749] LustreError: 123738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3742.992996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.153310] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3748.328678] LustreError: 123739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.418196] LustreError: 123738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3755.747338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.527512] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3766.789572] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3772.408120] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3779.999538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3787.129460] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3794.028350] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3800.492067] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3807.203485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3807.212648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.227339] Lustre: Skipped 1 previous similar message [ 3807.234270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.241059] Lustre: Skipped 4 previous similar messages [ 3810.882507] Lustre: server umount lustre-OST0000 complete [ 3817.442493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3817.469677] LustreError: Skipped 1 previous similar message [ 3823.050611] Lustre: server umount lustre-MDT0000 complete [ 3823.593767] LustreError: 123738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.626612] LustreError: 123738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3827.122905] LustreError: 123720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645231 with bad export cookie 16302360004817191635 [ 3827.126069] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3827.135138] LustreError: 123720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3827.600377] Lustre: server umount lustre-MDT0001 complete [ 3840.370142] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3843.419488] Key type lgssc unregistered [ 3843.755265] LNet: 127370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3843.761761] LNetError: 127370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3843.774208] LNet: Removed LNI 192.168.202.109@tcp [ 3844.572983] Key type .llcrypt unregistered [ 3844.575753] Key type ._llcrypt unregistered [ 3862.073931] Key type ._llcrypt registered [ 3862.076380] Key type .llcrypt registered [ 3862.151952] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3878.400771] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3880.166599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3880.453335] alg: No test for adler32 (adler32-zlib) [ 3881.935553] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3882.344606] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3884.088517] Key type lgssc registered [ 3885.719756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3924.792962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3924.804964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.015329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3926.048835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3926.172689] Lustre: lustre-MDT0000: new disk, initializing [ 3926.307794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.376704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3931.881584] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3942.483763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.641414] Lustre: 130523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3942.761191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3942.766278] Lustre: Skipped 1 previous similar message [ 3942.853684] Lustre: lustre-MDT0001: new disk, initializing [ 3942.968199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3943.019344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3943.037026] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3948.595309] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3958.779445] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3965.453880] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3975.311024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3975.614783] Lustre: lustre-OST0000: new disk, initializing [ 3975.617967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3975.624056] Lustre: 132495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3975.694701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.042684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3985.075577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3985.208584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3985.424363] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3995.730357] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4000.100634] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4000.399737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4006.453752] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4006.857079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4010.983838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.986265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.987475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.993739] Lustre: Skipped 1 previous similar message [ 4011.032715] LustreError: Skipped 1 previous similar message [ 4014.881681] Lustre: server umount lustre-OST0000 complete [ 4019.685401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4019.692288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.712565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.720177] Lustre: Skipped 1 previous similar message [ 4021.220240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.220481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.320794] Lustre: server umount lustre-MDT0000 complete [ 4027.134158] LustreError: 132494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645431 with bad export cookie 13175921707247631537 [ 4027.138059] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4027.146042] LustreError: 132494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4027.533295] Lustre: server umount lustre-MDT0001 complete [ 4035.012638] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 14:23:58 (1786645438) [ 4041.384311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.840439] LustreError: 134906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.865021] LustreError: 134906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4041.927924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.384072] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4047.330716] LustreError: 134907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.437147] LustreError: 134906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4053.571760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.950962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.914988] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4064.310701] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4070.242892] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4074.469148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4074.476041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.492226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.494806] Lustre: Skipped 1 previous similar message [ 4078.563978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4078.568365] Lustre: Skipped 3 previous similar messages [ 4079.149115] Lustre: server umount lustre-MDT0000 complete [ 4083.018299] LustreError: 134886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645487 with bad export cookie 13175921707247632426 [ 4083.025600] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4083.035203] LustreError: 134886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4083.357197] Lustre: server umount lustre-MDT0001 complete [ 4092.727371] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 14:24:56 (1786645496) [ 4100.535624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4116.064593] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4121.184670] LustreError: 137321:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 4146.912655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.344806] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4160.798463] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4163.164032] Lustre: server umount lustre-OST0000 complete [ 4171.968556] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 14:26:15 (1786645575) [ 4178.121481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4178.445453] LustreError: 138767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.494571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.415909] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4183.521808] LustreError: 138768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4190.843084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.170639] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4201.907209] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4207.494427] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4212.774892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4213.028932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.035217] Lustre: Skipped 1 previous similar message [ 4218.537434] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4225.268249] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4233.699576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.704233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4233.714755] Lustre: Skipped 2 previous similar messages [ 4233.725139] Lustre: Skipped 2 previous similar messages [ 4238.669654] Lustre: server umount lustre-MDT0000 complete [ 4238.821063] LustreError: 138763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.831955] LustreError: 138763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4241.757288] LustreError: 138748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645646 with bad export cookie 13175921707247633084 [ 4241.778241] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.061803] Lustre: server umount lustre-MDT0001 complete [ 4255.955337] Lustre: server umount lustre-OST0000 complete [ 4265.259716] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 14:27:47 (1786645667) [ 4272.937676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.804656] LustreError: 142419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.655846] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4290.947104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4291.517148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4291.529191] Lustre: Skipped 1 previous similar message [ 4296.984680] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4307.275535] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4313.563535] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4320.449896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4328.285897] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4337.115494] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4343.061372] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4343.348545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4344.473257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4350.781479] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4351.166669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4353.509828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.525872] Lustre: Skipped 2 previous similar messages [ 4353.526931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.689297] Lustre: server umount lustre-OST0000 complete [ 4368.356374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4371.759276] Lustre: server umount lustre-MDT0000 complete [ 4373.986023] LustreError: 142416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.003646] LustreError: 142416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4375.739024] LustreError: 142399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645780 with bad export cookie 13175921707247634155 [ 4375.757338] LustreError: 142399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4375.768536] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4376.192424] Lustre: server umount lustre-MDT0001 complete [ 4384.265666] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 14:29:47 (1786645787) [ 4391.214764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4406.882825] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4412.000494] LustreError: 146590:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 4437.728633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.743689] Lustre: Skipped 1 previous similar message [ 4444.505994] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4450.953596] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4457.619763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.882182] LustreError: 146615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.973697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4462.686978] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4471.365950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.574457] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4486.073409] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4493.658394] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4498.506943] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4498.732640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4503.590423] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4503.770306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4507.617852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4507.629931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.651721] Lustre: Skipped 2 previous similar messages [ 4507.661785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.668298] Lustre: Skipped 5 previous similar messages [ 4512.751970] Lustre: server umount lustre-MDT0000 complete [ 4516.628122] LustreError: 146597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645921 with bad export cookie 13175921707247635191 [ 4516.640209] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4516.655911] LustreError: 146597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4516.921631] Lustre: server umount lustre-MDT0001 complete [ 4529.522279] Lustre: server umount lustre-OST0000 complete [ 4536.699197] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 14:32:20 (1786645940) [ 4542.644616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4558.306911] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4563.424747] LustreError: 150782:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 4598.422849] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4608.211193] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4615.199230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4615.746830] LustreError: 150807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.784257] LustreError: 150807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4616.048819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4620.965179] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4634.378555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.833668] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4650.177528] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4656.724823] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4665.179272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4665.378374] Lustre: lustre-OST0001: new disk, initializing [ 4665.382357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4665.390512] Lustre: 153657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4670.817302] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4673.681696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4673.710377] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4673.972806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4681.478232] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4686.500854] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4686.740448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4692.119383] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4692.482138] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4696.043234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4696.075710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4696.098714] Lustre: Skipped 2 previous similar messages [ 4696.106207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4696.108990] Lustre: Skipped 3 previous similar messages [ 4709.344188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4709.513543] Lustre: server umount lustre-OST0000 complete [ 4714.984278] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4715.005485] LustreError: Skipped 2 previous similar messages [ 4720.409835] Lustre: server umount lustre-OST0001 complete [ 4731.898098] Lustre: server umount lustre-MDT0000 complete [ 4736.071437] LustreError: 150788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646140 with bad export cookie 13175921707247636143 [ 4736.076818] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.084529] LustreError: 150788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4736.490377] Lustre: server umount lustre-MDT0001 complete [ 4762.384579] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:36:06 (1786646166) [ 4764.182604] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4765.998407] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4767.417156] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:36:11 (1786646171) [ 4774.762786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.281348] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4775.323678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4775.751843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.768810] Lustre: Skipped 6 previous similar messages [ 4781.366660] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4793.442958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4793.706910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4793.785274] Lustre: 157221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4799.253300] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4806.732971] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4812.758722] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4823.404155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4823.662883] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4831.051519] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4831.124082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4842.757735] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4849.754448] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4849.886755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4856.188470] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4856.391980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4861.921030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4861.928643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.937335] LustreError: Skipped 2 previous similar messages [ 4861.940840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4861.940848] Lustre: Skipped 12 previous similar messages [ 4861.982905] Lustre: Skipped 6 previous similar messages [ 4864.572813] Lustre: server umount lustre-OST0000 complete [ 4888.251489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4896.657670] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4901.810232] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.838020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.848441] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4907.541772] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4914.835858] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4914.976214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4920.132430] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4920.389470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4925.888578] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95cf864dd000.ost_server_uuid 50 [ 4927.359334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95cf864dd000.ost_server_uuid in FULL state after 0 sec [ 4927.521862] LustreError: 157226:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.9@tcp) returned error from blocking AST (req@000000009c342915 x1873433368740224 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b13e7acb/0xb6da46fbd4da9809 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.9@tcp remote: 0x5318c248f822408c expref: 5 pid: 159091 timeout: 5027 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4927.541986] LustreError: lustre-MDT0000: A client on nid 192.168.202.9@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4927.548841] LustreError: 157216:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.9@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b13e7acb/0xb6da46fbd4da9809 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.9@tcp remote: 0x5318c248f822408c expref: 6 pid: 159091 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4932.587019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4932.606100] LustreError: Skipped 1 previous similar message [ 4937.511667] Lustre: server umount lustre-OST0000 complete [ 4948.388749] Lustre: server umount lustre-MDT0000 complete [ 4951.825132] LustreError: 157213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646356 with bad export cookie 13175921707247637326 [ 4951.832624] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4966.368344] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4966.775742] Lustre: server umount lustre-MDT0001 complete [ 4981.891877] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 4985.131505] Key type lgssc unregistered [ 4985.421523] LNet: 163343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4985.428651] LNetError: 163343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4985.443491] LNet: Removed LNI 192.168.202.109@tcp [ 4986.175256] Key type .llcrypt unregistered [ 4986.176913] Key type ._llcrypt unregistered [ 5010.424506] Key type ._llcrypt registered [ 5010.427360] Key type .llcrypt registered [ 5010.515233] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5011.722771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5011.737607] alg: No test for adler32 (adler32-zlib) [ 5012.796463] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5013.003806] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5014.648490] Key type lgssc registered [ 5015.691453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5023.244975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5023.261401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.567656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5028.554649] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5035.001695] LustreError: 164762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.021425] LustreError: 164762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5038.021560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.642308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5042.827545] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5049.952631] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5055.751685] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5063.319957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5063.874211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.055698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5073.263505] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5083.249188] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5089.959305] Lustre: Failing over lustre-MDT0000 [ 5090.190853] Lustre: server umount lustre-MDT0000 complete [ 5090.796910] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.805683] LustreError: 164758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5090.811944] Lustre: Skipped 2 previous similar messages [ 5090.825791] LustreError: 164758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5095.907958] LustreError: 167256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.946588] LustreError: 167256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5101.028242] LustreError: 164763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.044054] LustreError: 164763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5102.797420] LustreError: 164744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.9@tcp arrived at 1786646507 with bad export cookie 15266278124413026887 [ 5102.824248] LustreError: 164744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5106.145889] LustreError: 164762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.164904] LustreError: 164762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5107.104157] Lustre: 164075:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786646495/real 1786646495] req@00000000b2d3e30c x1873434553379200/t0(0) o400->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786646511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5107.124157] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5110.611241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.389219] LustreError: 167256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5116.398016] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd3dcb711e7f482a9 [ 5116.418805] LustreError: 167256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5116.444494] Lustre: MGC192.168.202.109@tcp: Connection restored to 0@lo (at 0@lo) [ 5116.904712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.966723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5122.023650] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5122.041679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5122.380454] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5124.838930] Lustre: lustre-MDT0000: Denying connection for new client e1627b47-a4b6-4206-aef6-e50591e10c38 (at 192.168.202.9@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5136.869329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5136.874593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.884927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5137.393919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5137.402398] Lustre: Skipped 1 previous similar message [ 5139.660898] Lustre: server umount lustre-OST0000 complete [ 5144.197222] LustreError: 168496:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5144.207665] Lustre: 167845:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5144.218943] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5144.235259] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5144.252145] LustreError: 167845:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000038508b0d x1873434553399936/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5144.274756] LustreError: 167845:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5144.284103] LustreError: 167845:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5144.293794] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5144.713606] Lustre: server umount lustre-MDT0000 complete [ 5147.620823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.642319] Lustre: Skipped 1 previous similar message [ 5147.650307] LustreError: 164757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.085542] LustreError: 164743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646553 with bad export cookie 15266278124413026985 [ 5149.087797] LustreError: 164072:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5149.098126] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5152.212156] Lustre: server umount lustre-MDT0001 complete [ 5164.297778] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5167.240984] Key type lgssc unregistered [ 5167.569674] LNet: 169251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5167.577845] LNetError: 169251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5167.592871] LNet: Removed LNI 192.168.202.109@tcp [ 5168.409809] Key type .llcrypt unregistered [ 5168.412368] Key type ._llcrypt unregistered [ 5190.533958] Key type ._llcrypt registered [ 5190.540048] Key type .llcrypt registered [ 5190.632563] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5191.683122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5191.718245] alg: No test for adler32 (adler32-zlib) [ 5192.749725] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5193.003844] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5194.688882] Key type lgssc registered [ 5195.665921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5202.828542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5202.850361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.190680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.084880] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5214.692781] LustreError: 170643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5214.703807] LustreError: 170643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5215.609973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.928136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5219.720718] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5225.432305] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5229.897732] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5234.819719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5235.100569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.781967] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5246.513988] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5249.461065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5259.758123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5259.758523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.809265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5259.819696] Lustre: Skipped 1 previous similar message [ 5264.875801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5264.878206] Lustre: Skipped 1 previous similar message [ 5269.995428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5269.999765] Lustre: Skipped 1 previous similar message [ 5271.008225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5271.116317] Lustre: server umount lustre-OST0000 complete [ 5275.109627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.112587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5275.126550] Lustre: Skipped 2 previous similar messages [ 5280.226075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5280.233813] Lustre: Skipped 2 previous similar messages [ 5280.546846] Lustre: server umount lustre-MDT0000 complete [ 5283.946212] LustreError: 173426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646688 with bad export cookie 3087200261792781832 [ 5283.947258] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5283.956402] LustreError: 173426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5284.182556] Lustre: server umount lustre-MDT0001 complete [ 5293.815437] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5296.222221] Key type lgssc unregistered [ 5296.585890] LNet: 174180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5296.590953] LNetError: 174180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5296.634167] LNet: Removed LNI 192.168.202.109@tcp [ 5297.462909] Key type .llcrypt unregistered [ 5297.466710] Key type ._llcrypt unregistered [ 5331.256488] Key type ._llcrypt registered [ 5331.266356] Key type .llcrypt registered [ 5331.437464] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5332.467993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5332.701414] alg: No test for adler32 (adler32-zlib) [ 5334.037651] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5334.492858] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5336.224290] Key type lgssc registered [ 5337.515834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5345.005807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5345.023838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.546909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5350.475216] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5357.025903] LustreError: 175884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5357.042246] LustreError: 175884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5359.518856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5359.777919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5363.414362] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5371.594793] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5378.939463] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5387.138632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5387.558425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.680442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5394.307090] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5401.907621] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5415.397488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.400043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5415.402505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5415.409795] Lustre: Skipped 1 previous similar message [ 5420.513296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5420.523471] Lustre: Skipped 1 previous similar message [ 5421.190614] Lustre: server umount lustre-OST0000 complete [ 5425.635934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.651616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5425.655191] Lustre: Skipped 2 previous similar messages [ 5430.753611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5431.503085] Lustre: server umount lustre-MDT0000 complete [ 5435.874149] LustreError: 175885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.889902] LustreError: 175885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5436.246408] LustreError: 175864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646841 with bad export cookie 13529604788384928548 [ 5436.258209] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5436.264240] LustreError: 175864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5436.541034] Lustre: server umount lustre-MDT0001 complete [ 5447.624976] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5450.774175] Key type lgssc unregistered [ 5451.146833] LNet: 179440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5451.158464] LNetError: 179440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5451.191920] LNet: Removed LNI 192.168.202.109@tcp [ 5452.288275] Key type .llcrypt unregistered [ 5452.290815] Key type ._llcrypt unregistered [ 5472.484432] Key type ._llcrypt registered [ 5472.487464] Key type .llcrypt registered [ 5472.573227] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5473.535202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5473.620681] alg: No test for adler32 (adler32-zlib) [ 5474.718472] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5474.892936] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5476.552206] Key type lgssc registered [ 5477.524599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5487.052649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5487.069782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5488.402711] Lustre: *** cfs_fail_loc=135, val=0*** [ 5488.473059] LustreError: 180943:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5488.480144] LustreError: 180943:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -2 [ 5488.487950] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5488.487950] [ 5488.501840] LustreError: MGC192.168.202.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5488.512162] LustreError: 180904:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5488.522180] LustreError: 180904:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5488.536309] LustreError: 180904:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5488.539804] LustreError: 180904:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5488.629672] Lustre: server umount lustre-MDT0000 complete [ 5488.632824] LustreError: 180904:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5494.548565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.837317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5498.589173] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5504.995502] LustreError: 181312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.012961] LustreError: 181312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5505.517809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.756736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5509.261616] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5516.239979] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5521.022900] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5524.453212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.462563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5524.465201] Lustre: Skipped 1 previous similar message [ 5524.491121] Lustre: Skipped 1 previous similar message [ 5529.573327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5529.578189] Lustre: Skipped 1 previous similar message [ 5530.595339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.608507] Lustre: Skipped 1 previous similar message [ 5535.715076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5535.721352] Lustre: Skipped 1 previous similar message [ 5537.248165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5537.391484] Lustre: server umount lustre-MDT0000 complete [ 5540.403953] LustreError: 181294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646945 with bad export cookie 12784759673957049573 [ 5540.417724] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.711691] Lustre: server umount lustre-MDT0001 complete [ 5548.914767] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5551.148662] Key type lgssc unregistered [ 5551.449652] LNet: 183636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5551.453928] LNetError: 183636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5551.468169] LNet: Removed LNI 192.168.202.109@tcp [ 5552.195829] Key type .llcrypt unregistered [ 5552.199407] Key type ._llcrypt unregistered [ 5569.968708] Key type ._llcrypt registered [ 5569.971373] Key type .llcrypt registered [ 5570.080645] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5570.412407] Key type .llcrypt unregistered [ 5570.414202] Key type ._llcrypt unregistered [ 5584.357846] Key type ._llcrypt registered [ 5584.359706] Key type .llcrypt registered [ 5584.454621] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5585.369715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5585.387198] alg: No test for adler32 (adler32-zlib) [ 5586.378332] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5586.546846] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5588.176683] Key type lgssc registered [ 5589.199660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5596.039655] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5596.077117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5611.616394] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5616.736294] LustreError: 185752:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 5643.425620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5649.133927] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5655.632964] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5663.436409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.552478] LustreError: 185087:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001d601b02 x1873435155237120/t0(0) o250->MGC192.168.202.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5663.953815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.999638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5668.962425] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5669.359546] LustreError: 185778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5669.374861] LustreError: 185778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5674.487423] LustreError: 185777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.608815] LustreError: 186787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.639285] LustreError: 186787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5678.499952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.731317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5683.920721] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5692.894513] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5698.583480] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5701.933871] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5715.426214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5715.437623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.454706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.037690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.042697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.042801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.488416] Lustre: server umount lustre-OST0000 complete [ 5725.155084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.159692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5725.162029] Lustre: Skipped 1 previous similar message [ 5725.167016] Lustre: Skipped 1 previous similar message [ 5730.273526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.295425] Lustre: Skipped 2 previous similar messages [ 5731.066403] Lustre: server umount lustre-MDT0000 complete [ 5734.567855] LustreError: 185758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647139 with bad export cookie 7763694221622402345 [ 5734.574516] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.592627] LustreError: 185758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5735.043863] Lustre: server umount lustre-MDT0001 complete [ 5745.858352] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5748.665314] Key type lgssc unregistered [ 5748.900166] LNet: 189813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5748.906695] LNetError: 189813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5748.938350] LNet: Removed LNI 192.168.202.109@tcp [ 5750.012101] Key type .llcrypt unregistered [ 5750.020083] Key type ._llcrypt unregistered [ 5774.704128] Key type ._llcrypt registered [ 5774.708460] Key type .llcrypt registered [ 5774.830409] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5775.780371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.939695] alg: No test for adler32 (adler32-zlib) [ 5777.101510] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5777.373881] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5779.072224] Key type lgssc registered [ 5779.883635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.614208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5786.651276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.306510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.979367] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5798.883351] LustreError: 191228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.897629] LustreError: 191228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5801.043941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.356232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5805.855528] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5813.116818] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5819.302695] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5825.927236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5826.284845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5831.864130] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5833.470054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5837.565411] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5842.089937] Lustre: Failing over lustre-MDT0000 [ 5842.411416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5842.428142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.459215] LustreError: 191227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.654688] Lustre: server umount lustre-MDT0000 complete [ 5843.960972] LustreError: 191228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.967475] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.063956] LustreError: 191224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.098446] LustreError: 191224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5854.179404] LustreError: 191223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.207000] LustreError: 191223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5859.296115] Lustre: 190538:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786647248/real 1786647248] req@0000000034821634 x1873435355537664/t0(0) o400->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786647264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5859.351746] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5862.835871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5864.424614] LustreError: 191222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.456734] LustreError: 191222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5869.541074] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x180d140d1a8dc7a9 [ 5869.549510] Lustre: MGC192.168.202.109@tcp: Connection restored to 0@lo (at 0@lo) [ 5869.864899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.000415] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5870.650511] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5874.219764] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5875.181471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5875.202364] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5875.252446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5878.913511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5889.213844] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5905.894850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.903827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.917363] Lustre: Skipped 2 previous similar messages [ 5905.940250] Lustre: Skipped 1 previous similar message [ 5911.011336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5911.033535] Lustre: Skipped 1 previous similar message [ 5911.535468] Lustre: server umount lustre-OST0000 complete [ 5921.250539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.254779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5921.265493] Lustre: Skipped 1 previous similar message [ 5921.268304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5921.284990] Lustre: Skipped 1 previous similar message [ 5922.826435] Lustre: server umount lustre-MDT0000 complete [ 5926.379097] LustreError: 193976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.400023] LustreError: 193976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5927.279517] LustreError: 191209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647332 with bad export cookie 1733063478120138665 [ 5927.291354] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5927.294366] LustreError: 191209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5927.828210] Lustre: server umount lustre-MDT0001 complete [ 5940.866479] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5944.405893] Key type lgssc unregistered [ 5944.856651] LNet: 196197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5944.868861] LNetError: 196197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5944.896420] LNet: Removed LNI 192.168.202.109@tcp [ 5945.685331] Key type .llcrypt unregistered [ 5945.687234] Key type ._llcrypt unregistered [ 5966.979706] Key type ._llcrypt registered [ 5966.981315] Key type .llcrypt registered [ 5967.111661] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5968.115756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5968.174057] alg: No test for adler32 (adler32-zlib) [ 5969.157350] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5969.311107] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5971.000173] Key type lgssc registered [ 5972.359356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5979.658497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5979.681875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.056637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5986.025905] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5990.391209] LustreError: 197591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.413150] LustreError: 197591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5995.493389] LustreError: 197592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5998.842609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.527177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6005.117471] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6012.926127] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6018.656451] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6026.747767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6027.261608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.164744] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6041.843972] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6050.848382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6056.391023] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6062.613442] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6083.164256] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6096.865608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6096.872546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6096.891676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6096.901104] Lustre: Skipped 1 previous similar message [ 6101.990858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6102.005416] Lustre: Skipped 2 previous similar messages [ 6102.094538] Lustre: server umount lustre-OST0000 complete [ 6107.114034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6107.120105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.149183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6107.170412] Lustre: Skipped 1 previous similar message [ 6107.191220] Lustre: Skipped 1 previous similar message [ 6112.227679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6112.765639] Lustre: server umount lustre-MDT0000 complete [ 6117.348279] LustreError: 197591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.373052] LustreError: 197591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6118.417617] LustreError: 197572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647523 with bad export cookie 17551443978218514017 [ 6118.434667] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6118.436203] LustreError: 197572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6119.245496] Lustre: server umount lustre-MDT0001 complete [ 6132.559743] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6135.812504] Key type lgssc unregistered [ 6136.103573] LNet: 201506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6136.109606] LNetError: 201506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6136.125331] LNet: Removed LNI 192.168.202.109@tcp [ 6136.944840] Key type .llcrypt unregistered [ 6136.949539] Key type ._llcrypt unregistered [ 6153.359466] Key type ._llcrypt registered [ 6153.361060] Key type .llcrypt registered [ 6153.466930] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6154.882834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6154.898089] alg: No test for adler32 (adler32-zlib) [ 6156.095982] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 6156.372873] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6158.048666] Key type lgssc registered [ 6159.207801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6165.925198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6165.960912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.483434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6172.610952] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6177.763593] LustreError: 202753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.779795] LustreError: 202753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6182.893410] LustreError: 202754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.091140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6183.514722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6187.668217] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6194.743942] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6201.819590] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6208.536035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6208.870679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.976373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6214.141372] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6220.469530] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6225.362585] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6233.677663] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6245.161995] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6250.026949] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6261.217155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6261.219845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.230874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6264.297331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6264.302364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6264.302373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6264.323050] Lustre: Skipped 1 previous similar message [ 6266.536957] Lustre: server umount lustre-OST0000 complete [ 6274.531865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.544842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6274.549075] Lustre: Skipped 1 previous similar message [ 6276.806711] Lustre: server umount lustre-MDT0000 complete [ 6279.652491] LustreError: 203481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.675378] LustreError: 203481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6280.502863] LustreError: 204601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647685 with bad export cookie 18233975368510460725 [ 6280.506415] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6280.526874] LustreError: 204601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6280.809707] Lustre: server umount lustre-MDT0001 complete [ 6295.680162] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6298.465289] Key type lgssc unregistered [ 6298.766422] LNet: 207696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6298.770402] LNetError: 207696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6298.817955] LNet: Removed LNI 192.168.202.109@tcp [ 6299.441495] Key type .llcrypt unregistered [ 6299.443357] Key type ._llcrypt unregistered [ 6327.985254] Key type ._llcrypt registered [ 6327.988416] Key type .llcrypt registered [ 6328.088737] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6329.202808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6329.250731] alg: No test for adler32 (adler32-zlib) [ 6330.501983] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 6331.185612] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6332.977157] Key type lgssc registered [ 6334.166031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6344.892453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6344.927282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.100120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6354.060170] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6356.474693] LustreError: 209109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.485050] LustreError: 209109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6361.582700] LustreError: 209110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.692061] LustreError: 209109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.831080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.536310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6372.708875] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6380.151430] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6387.177388] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6394.574765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6395.325791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6402.375446] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6411.937849] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.871357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6433.257417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6433.261966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6433.272217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6438.377962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6438.397591] Lustre: Skipped 2 previous similar messages [ 6439.493889] Lustre: server umount lustre-OST0000 complete [ 6443.493778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6443.504522] Lustre: Skipped 1 previous similar message [ 6443.509420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6443.519424] Lustre: Skipped 1 previous similar message [ 6448.615063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6448.627516] Lustre: Skipped 1 previous similar message [ 6449.836563] Lustre: server umount lustre-MDT0000 complete [ 6453.730208] LustreError: 211604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6453.758562] LustreError: 211604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6454.388468] LustreError: 209090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647859 with bad export cookie 12106963116340629605 [ 6454.404156] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6454.832687] Lustre: server umount lustre-MDT0001 complete [ 6466.514949] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6469.730275] Key type lgssc unregistered [ 6469.988571] LNet: 212818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6469.994626] LNetError: 212818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6470.008303] LNet: Removed LNI 192.168.202.109@tcp [ 6470.828749] Key type .llcrypt unregistered [ 6470.837300] Key type ._llcrypt unregistered [ 6492.846786] Key type ._llcrypt registered [ 6492.849245] Key type .llcrypt registered [ 6492.935623] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6493.633613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6493.641169] alg: No test for adler32 (adler32-zlib) [ 6495.060907] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 6495.461547] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6497.152926] Key type lgssc registered [ 6498.576483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6508.501948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6508.535653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.156478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6515.294678] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6519.272752] LustreError: 214089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.326949] LustreError: 214089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6524.394379] LustreError: 214090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6526.080273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.622286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6534.563846] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6544.272430] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6550.956534] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6557.262192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6557.707155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.572050] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6570.398517] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6579.210695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6594.536395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6594.545720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6594.553220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6594.578859] Lustre: Skipped 1 previous similar message [ 6594.614306] Lustre: Skipped 1 previous similar message [ 6599.653930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6599.663951] Lustre: Skipped 1 previous similar message [ 6600.537413] Lustre: server umount lustre-OST0000 complete [ 6604.779183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6604.787732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6604.798361] Lustre: Skipped 1 previous similar message [ 6604.813923] Lustre: Skipped 1 previous similar message [ 6609.890559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.529876] Lustre: server umount lustre-MDT0000 complete [ 6614.082968] LustreError: 214070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648018 with bad export cookie 6461044410989711983 [ 6614.084103] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6614.091919] LustreError: 214070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6614.522757] Lustre: server umount lustre-MDT0001 complete [ 6625.160459] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6627.957466] Key type lgssc unregistered [ 6628.332409] LNet: 217800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6628.337861] LNetError: 217800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6629.354604] LNet: Removed LNI 192.168.202.109@tcp [ 6630.446406] Key type .llcrypt unregistered [ 6630.448535] Key type ._llcrypt unregistered [ 6652.364327] Key type ._llcrypt registered [ 6652.377063] Key type .llcrypt registered [ 6652.626352] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6654.182615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6654.319062] alg: No test for adler32 (adler32-zlib) [ 6655.434459] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 6655.845565] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6657.585070] Key type lgssc registered [ 6658.962106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6667.472925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6667.502783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6669.201255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.047329] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6678.501497] LustreError: 219070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6678.513294] LustreError: 219070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6683.622113] LustreError: 219071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.860714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6684.548873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6690.694873] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6701.910540] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6708.461302] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6717.515368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.242601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.064212] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6735.554267] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6738.753460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6748.185500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6748.235570] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6748.399024] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6755.398466] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6765.328523] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6767.829908] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6780.450869] Lustre: Permanently deactivating lustre-OST0001 [ 6780.486253] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6787.706259] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6792.616315] Lustre: Permanently reactivating lustre-OST0001 [ 6792.628349] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6792.632826] Lustre: Skipped 2 previous similar messages [ 6797.954129] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.976585] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6798.000489] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6798.047482] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6798.112229] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6803.427378] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6803.434562] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.435855] LustreError: Skipped 1 previous similar message [ 6803.448673] Lustre: Skipped 1 previous similar message [ 6803.449579] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6807.020291] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6807.032026] Lustre: Skipped 2 previous similar messages [ 6809.490773] Lustre: server umount lustre-OST0001 complete [ 6812.151138] LustreError: 220892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.187062] LustreError: 220892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6815.714080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6815.730929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.754323] Lustre: Skipped 1 previous similar message [ 6815.773222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6816.737492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6817.252503] LustreError: 220892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.287492] LustreError: 220892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6821.289832] Lustre: server umount lustre-OST0000 complete [ 6827.489749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6827.492664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6827.525116] Lustre: Skipped 2 previous similar messages [ 6827.545828] Lustre: Skipped 4 previous similar messages [ 6832.610815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6832.625924] Lustre: Skipped 1 previous similar message [ 6839.264467] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6839.681535] Lustre: server umount lustre-MDT0000 complete [ 6842.863745] LustreError: 219070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.903194] LustreError: 219070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6844.543059] LustreError: 219052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648249 with bad export cookie 1071464583038330126 [ 6844.546520] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6844.566831] LustreError: 219052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6845.252816] Lustre: server umount lustre-MDT0001 complete [ 6857.521755] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6860.487606] Key type lgssc unregistered [ 6860.791969] LNet: 224123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6860.797372] LNetError: 224123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6861.868910] LNet: Removed LNI 192.168.202.109@tcp [ 6862.859806] Key type .llcrypt unregistered [ 6862.861946] Key type ._llcrypt unregistered [ 6900.543712] Key type ._llcrypt registered [ 6900.547330] Key type .llcrypt registered [ 6900.626617] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6901.792866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6901.966727] alg: No test for adler32 (adler32-zlib) [ 6903.332889] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 6903.618064] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6905.392912] Key type lgssc registered [ 6907.094662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6916.185935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6916.202221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.442260] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6917.483455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6917.665752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.183800] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6934.276438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.443513] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6934.515956] Lustre: 226322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6934.976409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6940.968040] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6951.463665] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6959.843803] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6970.161144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.390138] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6970.734119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6977.231280] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6978.217046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6985.063688] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6988.120913] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6992.098198] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7000.575830] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7008.526301] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7018.152823] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7038.112316] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7038.118909] Lustre: Skipped 2 previous similar messages [ 7070.138167] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7070.159803] Lustre: Skipped 3 previous similar messages [ 7134.789266] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7134.794839] Lustre: Skipped 8 previous similar messages [ 7157.217953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7157.230166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.267767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7157.734898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7157.748170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.781022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7159.607490] Lustre: server umount lustre-OST0000 complete [ 7165.409628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7165.421759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.440452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7167.970559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7167.970684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.506731] Lustre: server umount lustre-MDT0000 complete [ 7173.198275] LustreError: 229060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648578 with bad export cookie 37298442064129635 [ 7173.200241] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7173.205193] LustreError: 229060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7173.527907] Lustre: server umount lustre-MDT0001 complete [ 7185.380509] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7188.532041] Key type lgssc unregistered [ 7188.911553] LNet: 231130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7188.918466] LNetError: 231130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7188.949168] LNet: Removed LNI 192.168.202.109@tcp [ 7189.836638] Key type .llcrypt unregistered [ 7189.839493] Key type ._llcrypt unregistered [ 7214.265651] Key type ._llcrypt registered [ 7214.268487] Key type .llcrypt registered [ 7214.349039] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7215.535462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7215.563744] alg: No test for adler32 (adler32-zlib) [ 7216.555229] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 7216.776272] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7218.448521] Key type lgssc registered [ 7219.659459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7227.825604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7227.839316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7229.244080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.829471] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7239.654346] LustreError: 232541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.681518] LustreError: 232541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7244.770152] LustreError: 232540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.412681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.090198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7252.563961] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7262.079828] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7270.393195] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7278.636631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7279.122892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.275490] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7294.539903] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7299.652847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7316.962367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7316.973286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7316.991223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7320.042215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7320.050906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7320.060661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.164917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.175242] Lustre: Skipped 1 previous similar message [ 7329.249096] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7329.767171] Lustre: server umount lustre-OST0000 complete [ 7338.979304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7338.993565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.009726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7339.024284] Lustre: Skipped 1 previous similar message [ 7341.777317] Lustre: server umount lustre-MDT0000 complete [ 7345.633442] LustreError: 232541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.669485] LustreError: 232541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7346.683408] LustreError: 232522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648751 with bad export cookie 2504061989293841253 [ 7346.692947] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7346.708829] LustreError: 232522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7347.128806] Lustre: server umount lustre-MDT0001 complete [ 7361.820972] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7366.123768] Key type lgssc unregistered [ 7366.627039] LNet: 236237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7366.637038] LNetError: 236237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7366.661737] LNet: Removed LNI 192.168.202.109@tcp [ 7367.983164] Key type .llcrypt unregistered [ 7367.987801] Key type ._llcrypt unregistered [ 7398.450723] Key type ._llcrypt registered [ 7398.455769] Key type .llcrypt registered [ 7398.581954] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7400.463042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7400.505370] alg: No test for adler32 (adler32-zlib) [ 7401.893313] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 7402.310853] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7404.136205] Key type lgssc registered [ 7405.747931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7680.457806] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7684.720795] Key type lgssc unregistered [ 7685.088715] LNet: 238268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7685.112615] LNetError: 238268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7685.127844] LNet: Removed LNI 192.168.202.109@tcp [ 7686.253667] Key type .llcrypt unregistered [ 7686.255777] Key type ._llcrypt unregistered [ 7717.206189] Key type ._llcrypt registered [ 7717.207810] Key type .llcrypt registered [ 7717.374155] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 7738.954396] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7739.631990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7739.684318] alg: No test for adler32 (adler32-zlib) [ 7740.797624] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 7740.999313] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7742.776262] Key type lgssc registered [ 7744.243831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7781.206912] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7502 sec ========== 15:26:24 (1786649184) [ 7782.936346] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:26:26 (1786649186) === [ 7786.203548] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:26:29 (1786649189) === [ 7832.963505] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7835.839619] Key type lgssc unregistered [ 7836.189316] LNet: 243509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7836.197288] LNetError: 243509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7836.254492] LNet: Removed LNI 192.168.202.109@tcp [ 7837.192506] Key type .llcrypt unregistered [ 7837.194786] Key type ._llcrypt unregistered